STAC GIS AI Agent — Architecture Documentation

A deep-dive guide for developers on how the AI assistant works, from the React frontend through the FastAPI backend down to MCP tools and the Ollama LLM.

System Overview
High-Level Architecture Diagram
Frontend Layer
API Layer — FastAPI Chat Endpoints
Session & Concurrency Management
Agent Orchestration — The ReAct Loop
Memory System
MCP Server & Tool Registry
MCP Client — Direct Mode
LLM Provider — Ollama
MCP Tools Reference
MCP Resources & Prompts
Data Flow — Step by Step
Key Design Decisions

System Overview

The AI assistant is a ReAct-style (Reason + Act) agentic system that connects a React/TypeScript frontend to a locally-hosted Ollama LLM via a FastAPI backend. The agent can reason through problems, call geospatial tools, and return rich answers — all streamed token-by-token to the browser.

Tech stack at a glance:

Layer	Technology
Frontend	React + TypeScript (Vite, Zustand)
API	FastAPI (Python, async)
Agent Orchestrator	Custom ReAct loop (`agent_loop.py`)
LLM	Ollama (`qwen3.5` or similar)
Tool Protocol	MCP (Model Context Protocol) via `FastMCP`
Short-term Memory	Redis (conversation sessions)
Long-term Memory	PostgreSQL + pgvector (semantic search)
Embeddings	Ollama `nomic-embed-text`

High-Level Architecture Diagram

┌─────────────────────────────────────────────────────────┐
│                     BROWSER (React)                      │
│                                                         │
│  ChatBox.tsx ──► useChat hook ──► /api/v1/chat/stream   │
│  AgentActivityFeed.tsx  (SSE events rendered live)      │
└────────────────────────┬────────────────────────────────┘
                         │  HTTP POST + SSE stream
                         ▼
┌─────────────────────────────────────────────────────────┐
│                   FastAPI Backend                        │
│                                                         │
│  POST /api/v1/chat/stream                               │
│    ├── ConcurrencyGate (Redis Sorted Set)               │
│    ├── RedisSessionStore (conversation memory)          │
│    └── creates AgentLoop + MCPClient + OllamaProvider   │
└────────────────────────┬────────────────────────────────┘
                         │  in-process (no HTTP)
                         ▼
┌─────────────────────────────────────────────────────────┐
│              AgentLoop (ReAct Orchestrator)              │
│                                                         │
│  THINK ──► ACT ──► OBSERVE ──► THINK ──► RESPOND        │
│                                                         │
│  ├── calls OllamaProvider.chat_stream()                 │
│  ├── parses tool calls from LLM output                  │
│  └── calls MCPClient.call_tool()                        │
└────────────────────────┬────────────────────────────────┘
                         │  in-memory MCP protocol
                         ▼
┌─────────────────────────────────────────────────────────┐
│                   MCP Server (FastMCP)                   │
│                                                         │
│  Tools: raster, vector, analytics, memory, knowledge    │
│  Resources: collections://, vector://, memory://        │
│  Prompts: analysis_prompts, query_prompts               │
│                                                         │
│  All tools share: DB session, services, LLM embed fn    │
└─────────────────────────────────────────────────────────┘

Frontend Layer

Key files:

src/components/chat/ChatBox.tsx — main chat UI component
src/components/chat/AgentActivityFeed.tsx — live reasoning log
src/components/chat/MessageBubble.tsx — individual message renderer
src/hooks/use-chat.ts — all chat state + SSE communication
src/api/endpoints/chat.ts — raw API calls

`useChat` Hook

This is the central brain of the frontend. It:

Sends a message via streamChatMessage() → opens an EventSource-style SSE connection to /api/v1/chat/stream.
Handles SSE events as they arrive:
- session → stores the session ID in localStorage for continuity
- thinking → appends a thinking step to activities[]
- tool_start → appends tool invocation to activities[]
- tool_result → appends tool completion + duration to activities[]
- tool_error → appends error entry to activities[]
- content → streams text into the assistant message bubble in real time
- done → finalizes the message, stops loading indicator
Manages queue state: if the server is at max concurrent capacity (503 response), the hook enters isQueued mode and polls /api/v1/chat/capacity every 5 seconds, auto-retrying when a slot opens.
Session management: loadSession, deleteSession, renameSession, refreshSessions — all backed by API calls to session endpoints.

`ChatBox` Component

Responsibilities:

Renders the message list via <MessageBubble> components
Shows the <AgentActivityFeed> while the agent is thinking
Handles voice input via the Web Speech API (SpeechRecognition)
Handles TTS output via useTextToSpeech (browser speech synthesis)
Map bridge: watches activities for specific tool calls (visualize_collection, zoom_to_location, set_map_style, toggle_ui_panel) and fires callback props to the parent map component — this is how the agent controls the map.
Passes location (map center) and selectedFeature (clicked GeoJSON feature) with every message so the agent has spatial context.

`AgentActivityFeed` Component

A live scrollable log that shows:

Thinking(N) — the agent's internal reasoning text, streamed in italics
Calling <tool_name> — when a tool is being invoked
<tool_name> completed (Xms) — tool success with duration
Error entries with red styling

The feed auto-scrolls to the bottom as new steps arrive but detects if the user has scrolled up and stops auto-scrolling.

API Layer

File: backend/stacgis/app/api/v1/endpoints/chat.py

Endpoints

Method	Path	Description
`POST`	`/api/v1/chat/`	Synchronous — waits for full response
`POST`	`/api/v1/chat/stream`	Primary — SSE streaming response
`GET`	`/api/v1/chat/capacity`	Returns active/max/available slots
`GET`	`/api/v1/chat/sessions`	Lists sessions for logged-in user
`DELETE`	`/api/v1/chat/session/{id}`	Deletes from Redis + DB
`GET`	`/api/v1/chat/session/{id}/history`	Returns chat history

Streaming Endpoint Flow (`POST /chat/stream`)

# 1. Get or create session from Redis
session_id, session = await session_store.get_or_create(request.session_id)

# 2. Acquire concurrency slot
await concurrency_gate.acquire(session_id)  # blocks if at capacity

# 3. Build agent stack
agent, mcp_client, llm = await create_agent(memory, db, user_id)

# 4. Stream AgentStep objects and convert to SSE
async for step in agent.run_streaming(message, location, feature):
    yield sse_format(event_name, data)

# 5. Save to Redis + DB, release slot, close connections

Each AgentStep maps to one of these SSE events:

AgentState	SSE Event	Payload
`THINKING`	`thinking`	`{step, content}`
`ACTING`	`tool_start`	`{step, tool_name, tool_args}`
`OBSERVING`	`tool_result`	`{step, tool_name, tool_result, duration_ms}`
`ERROR`	`tool_error`	`{step, tool_name, error}`
`RESPONDING`	`content`	`{text, is_complete}`
—	`done`	`{session_id, message_id, content, timestamp}`

Session & Concurrency Management

`RedisSessionStore`

Each conversation is a Redis key session:{uuid} with a 1-hour TTL
Stores a serialized ConversationMemory object (message history)
The TTL is refreshed on every message, so active sessions never expire
On first message: a new UUID is generated and returned as the session_id

`ConcurrencyGate`

Prevents the server from being overwhelmed by too many simultaneous LLM calls.

Backed by a Redis Sorted Set (chat:active_slots)
Key = session_id, score = Unix timestamp
acquire(): checks current count vs CHAT_MAX_CONCURRENT_USERS; adds the session if under limit
refresh(): called periodically during streaming to reset the TTL score
release(): removes the session from the set when the stream ends
Stale entries (older than CHAT_SLOT_TTL_SECONDS) are pruned on every operation — so a crashed server never permanently blocks slots

Agent Orchestration — The ReAct Loop

File: backend/stacgis/app/agent/orchestrator/agent_loop.py

The AgentLoop class implements the classic ReAct pattern:

THINK → ACT → OBSERVE → THINK → ACT → OBSERVE → ... → RESPOND

Initialization (`initialize()`)

Before the first message, the agent:

Connects the MCPClient (in-memory, no network)
Builds the system prompt — includes the DEFAULT_SYSTEM_PROMPT plus:
- Step limit constraint
- User ID context (for memory tool)
- Pre-loaded list of all raster collections (collections://list)
- Pre-loaded list of all vector datasets (vector://datasets/all)
Embeds every tool using llm.embed() — these embeddings are used later for dynamic tool selection

Per-Step Loop (streaming variant: `run_streaming`)

On every iteration:

Step 1 — Dynamic Tool Selection

query_embedding = await llm.embed(f"{user_message} {last_thought}")
relevant_tools = mcp.get_relevant_tool_schemas(query_embedding, top_k=20)

Instead of giving all tools to the LLM every turn (which wastes tokens), the agent uses cosine similarity + keyword overlap to select the top-K most relevant tools for this specific step. This allows scaling to many tools without blowing up the context window.

Step 2 — LLM Call (streamed)

async for chunk in llm.chat_stream(messages, tools=ollama_tools):
    if chunk.type == "thinking":   yield ThinkingStep
    elif chunk.type == "content":  yield RespondingStep
    elif chunk.type == "done":     response = chunk.response

The Ollama stream yields three chunk types:

thinking — the model's chain-of-thought (from <think> tags or dedicated thinking field)
content — the actual response text
done — signals end of stream with the full parsed response

Step 3 — Decision Branch

Branch A: Tool call detected

for tc in response.tool_calls:
    validate_tool_call(tc, mcp.tool_schemas)  # schema check
    result = await mcp.call_tool(tc.name, tc.arguments)
    memory.add_tool_result(tc.id, tc.name, result)
    yield ObservingStep
continue  # loop back to THINK

Branch B: No tool calls → final response

# Strip <thought>, <tool_call> tags from content
# Yield clean final text
yield RespondingStep(is_complete=True)
return

Safety Guards

Step limit: configurable max_steps (default 15). One step before the limit, the agent receives a CRITICAL INSTRUCTION forcing it to summarize with no more tool calls.
Context length check: token count is estimated before every LLM call; raises an error if it would exceed num_ctx.
Tool result truncation: results longer than 10,000 chars are truncated before being added to memory.

`AgentState` Enum

class AgentState(Enum):
    IDLE       = "idle"
    THINKING   = "thinking"   # LLM is generating
    ACTING     = "acting"     # tool call sent
    OBSERVING  = "observing"  # tool result received
    RESPONDING = "responding" # final answer
    ERROR      = "error"      # tool failure

Memory System

The agent has two levels of memory:

1. Short-term: `ConversationMemory`

File: backend/stacgis/app/agent/orchestrator/memory.py

In-memory (and Redis-persisted) sliding window of the last 50 messages. Features:

Stores system, user, assistant, and tool role messages
Pair preservation: never cuts in the middle of a tool-call / tool-result pair when truncating
Tool result truncation: results > 10,000 chars are trimmed with a notice
Serializes to dict for Redis storage via to_dict() / from_dict()

2. Long-term: `MemoryService` (pgvector)

File: backend/stacgis/app/domain/services/memory_service.py

Persistent facts/preferences stored in PostgreSQL with pgvector embeddings:

add_memory(user_id, content, node_type) — auto-embeds and stores
get_relevant_memories(user_id, query_embedding, limit=5) — L2-distance semantic search
Injected at the start of every request via MemoryService(db, embedding_provider=llm.embed)

Memory flow per request:

On incoming message → embed the query → fetch top-5 semantically similar past memories
Inject them into the conversation as a system message: "Relevant long-term memories..."
The agent can call the add_memory MCP tool to explicitly save new facts for future sessions

MCP Server & Tool Registry

File: backend/stacgis/app/mcp/server.py

The MCP server is built with FastMCP. It runs in the same process as the FastAPI app — no separate server needed.

Lifecycle (`server_lifespan`)

On first connection, the MCP server instantiates all backend services and makes them available via ctx.request_context.lifespan_context:

ServerDependencies(
    db_session,
    user_service,
    tile_service,
    timeseries_service,
    raster_service,
    analytics_service,
    catalog_service,
    colormap_service,
    memory_service,
    vector_service,
    knowledge_service,
    vector_repository,
    embedding_provider,   # llm.embed function
)

Every tool gets access to this via get_deps(ctx).

Tool Registration

Tools are registered by importing their modules. The @mcp.tool() decorator fires at import time:

tool_modules = [
    "app.mcp.tools.knowledge_tools",
    "app.mcp.tools.raster_tools",
    "app.mcp.tools.analytics_tools",
    "app.mcp.tools.vector_tools",
    "app.mcp.tools.discovery",
    "app.mcp.tools.memory_tools",
    # ... resources and prompts ...
]
for module_name in tool_modules:
    __import__(module_name)

MCP Client — Direct Mode

File: backend/stacgis/app/agent/client.py

The MCPClient connects to the MCP server in-process using paired in-memory streams (anyio.create_memory_object_stream), bypassing HTTP entirely. This means:

Zero network latency for tool calls
Full MCP protocol compliance (JSON-RPC, lifecycle, context)
The server runs its full server_lifespan in a background task group

Client  ──write──►  [in-memory stream A]  ──read──►  Server
Client  ◄──read──   [in-memory stream B]  ◄──write──  Server

Dynamic Tool Selection (`get_relevant_tool_schemas`)

At each reasoning step, instead of giving the LLM all tools, the client scores every tool:

score = cosine_similarity(query_embedding, tool_embedding)
       + keyword_overlap_count * 0.1

Top-K tools (default 20) are returned for that step. This keeps the prompt small and focuses the LLM.

LLM Provider — Ollama

File: backend/stacgis/app/agent/llm/ollama_provider.py

Communicates with a locally-running Ollama instance via httpx.AsyncClient.

Key Methods

Method	Description
`chat(messages, tools)`	Non-streaming completion
`chat_stream(messages, tools)`	Async generator of `StreamChunk` objects
`embed(text)`	Generates embedding vector via `/api/embed`
`format_tools(mcp_tools)`	Converts MCP schemas → Ollama tool format
`_sanitize_schema(schema)`	Strips `anyOf`, `oneOf`, `title`, etc. that Ollama doesn't support
`_sanitize_messages_for_ollama(messages)`	Fixes `tool_calls[].function.arguments` (must be dict, not JSON string)

Thinking Model Support

The provider handles reasoning models (e.g. Qwen3) that emit chain-of-thought in two ways:

Dedicated thinking field in the Ollama response message
<think>...</think> tags embedded in the content field

Both are normalized to <thought>...</thought> tags, which the agent loop then strips from the final user-visible response.

Schema Sanitization

Ollama's tool parser rejects certain JSON Schema features. The _sanitize_schema method recursively cleans schemas:

anyOf / oneOf → flattened to the first non-null type
title fields → removed
default: null → removed
Empty required: [] → removed

MCP Tools Reference

Raster Tools (`raster_tools.py`)

Tool	Description
`list_raster_collections`	Lists all geospatial raster datasets with optional bbox/time filters
`get_raster_info`	Returns metadata, variables, dimensions, CRS for a collection
`get_raster_statistics`	Computes min/max/mean/std for a bbox region
`get_timeseries`	Extracts time-series values at a lat/lon point
`query_raster_point`	Gets pixel values at a specific location
`query_multiple_rasters_point`	Queries multiple collections at one point
`search_raster_collections`	Keyword search across collection names/descriptions
`visualize_collection`	UI trigger — signals frontend to show a collection on the map
`zoom_to_location`	UI trigger — signals frontend to fly the map to lat/lon
`set_map_style`	UI trigger — changes the basemap style
`toggle_ui_panel`	UI trigger — opens/closes UI panels

Analytics Tools (`analytics_tools.py`)

Tool	Description
`generate_chart`	Creates Chart.js config for line/bar/scatter/pie charts
`compute_zonal_stats`	Area-weighted statistics for a polygon
`detect_change`	Compares two time periods to detect change

Vector Tools (`vector_tools.py`)

Tool	Description
`list_vector_datasets`	Lists all vector datasets
`search_vector_features`	Semantic search over vector features
`get_vector_feature`	Gets a specific feature by ID
`query_vector_by_bbox`	Spatial query within a bounding box

Knowledge Tools (`knowledge_tools.py`)

Tool	Description
`search_knowledge`	Semantic search over the knowledge base
`add_knowledge`	Adds new knowledge entries

Memory Tools (`memory_tools.py`)

Tool	Description
`add_memory`	Persists a fact/preference to pgvector long-term memory
`search_memories`	Semantic search over stored memories

Discovery Tool (`discovery.py`)

Tool	Description
`list_available_tools`	Returns all registered MCP tools with descriptions

MCP Resources & Prompts

Resources

Resources are read-only data endpoints the agent can pre-load:

URI	Description
`collections://list`	Full list of all raster collections (pre-loaded into system prompt)
`vector://datasets/all`	All vector datasets
`memory://recent`	Most recent memories for the user
`analytics://summary`	System analytics summary

Prompts

Prompt templates the agent can invoke:

Name	Description
`analysis_prompt`	Template for geospatial analysis tasks
`query_prompt`	Template for data query tasks

Data Flow — Step by Step

Here is the complete journey of a single user message:

1. User types "Show me NDVI statistics for Germany in 2023"
   └─► ChatBox.tsx calls sendMessage(text, location, selectedFeature)

2. useChat hook:
   └─► Creates user + assistant (empty, streaming=true) messages in state
   └─► Calls streamChatMessage() → POST /api/v1/chat/stream

3. FastAPI endpoint:
   └─► Gets/creates Redis session → loads ConversationMemory
   └─► Acquires concurrency slot in Redis sorted set
   └─► Calls create_agent() → builds MCPClient + OllamaProvider + AgentLoop

4. AgentLoop.run_streaming():
   └─► Embeds user message → fetches top-5 long-term memories (pgvector)
   └─► Injects memories as system message
   └─► Enters ReAct loop

5. Step 1 — THINKING:
   └─► Embeds "user_message + last_thought" → selects top-20 relevant tools
   └─► Calls OllamaProvider.chat_stream(messages, tools)
   └─► Streams <think>... tokens → yields THINKING AgentSteps
       └─► FastAPI SSE: event: thinking, data: {step:1, content:"..."}
           └─► useChat: updates activities[] with thinking step
               └─► AgentActivityFeed renders "Thinking(1): ..."

6. Step 1 — ACTING:
   └─► LLM emits tool call: list_raster_collections(bbox="5.8,47.3,15.0,55.1")
   └─► AgentLoop validates schema → yields ACTING step
       └─► FastAPI SSE: event: tool_start
           └─► useChat: activities[] += {type:"tool_start", toolName:...}
               └─► AgentActivityFeed: "Calling list_raster_collections"

7. Step 1 — OBSERVING:
   └─► MCPClient.call_tool() → in-memory → MCP server → list_raster_collections()
       └─► catalog_service.list_collections(bbox=[5.8,47.3,15,55.1])
   └─► Result returned as string → added to ConversationMemory
       └─► FastAPI SSE: event: tool_result, data: {duration_ms: 234}
           └─► AgentActivityFeed: "list_raster_collections completed (234ms)"

8. Steps 2-N — More tool calls (e.g. get_raster_statistics):
   └─► Same THINK→ACT→OBSERVE cycle repeated

9. Final step — RESPONDING:
   └─► LLM generates final answer (no tool calls)
   └─► <thought> and <tool_call> tags stripped from content
   └─► Yields RESPONDING step with is_complete=True
       └─► FastAPI SSE: event: content, data: {text: "...", is_complete: true}
           └─► useChat: updates assistant message content (streaming text renders)
       └─► FastAPI SSE: event: done
           └─► useChat: sets isStreaming=false, isLoading=false

10. Cleanup:
    └─► ConversationMemory saved to Redis
    └─► Message saved to PostgreSQL (if user logged in)
    └─► Usage tracked in UsageService
    └─► ConcurrencyGate slot released
    └─► OllamaProvider + MCPClient connections closed

Key Design Decisions

1. In-Process MCP (Direct Mode)

The MCP client connects via in-memory streams rather than HTTP. This eliminates network latency for every tool call and avoids "Session terminated" errors from stateless HTTP MCP transports.

2. Dynamic Tool Selection

Embedding every tool and doing cosine similarity per step means the agent can have 50+ tools registered without polluting the LLM context. Only the most relevant tools for each reasoning step are included.

3. Streaming-First Architecture

Both the LLM output and the agent steps are streamed. The frontend shows the agent's reasoning in real time, making the experience feel transparent and responsive even for multi-step queries that take 30+ seconds.

4. UI-Controllable Tools

Tools like visualize_collection, zoom_to_location, set_map_style don't actually do anything on the backend — they return simple success strings. The real action happens because ChatBox.tsx watches the activities stream and fires callback props to the map when these tools are called. This keeps the tool protocol stateless while enabling rich UI interactions.

5. Two-Level Memory

Short-term memory (Redis) keeps the conversation coherent within a session. Long-term memory (pgvector) lets the agent remember user preferences and facts across sessions. The embedding model (nomic-embed-text) is the same one used for tool selection, keeping dependencies minimal.

6. Concurrency Control via Redis Sorted Set

Using scores as timestamps allows the gate to self-heal: if a server crashes mid-stream, the stale slot is automatically pruned after CHAT_SLOT_TTL_SECONDS without needing a cleanup job.

Table of Contents​

System Overview​

High-Level Architecture Diagram​

Frontend Layer​

useChat Hook​

ChatBox Component​

AgentActivityFeed Component​

API Layer​

Endpoints​

Streaming Endpoint Flow (POST /chat/stream)​

Session & Concurrency Management​

RedisSessionStore​

ConcurrencyGate​

Agent Orchestration — The ReAct Loop​

Initialization (initialize())​

Per-Step Loop (streaming variant: run_streaming)​

Step 1 — Dynamic Tool Selection​

Step 2 — LLM Call (streamed)​

Step 3 — Decision Branch​

Safety Guards​

AgentState Enum​

Memory System​

1. Short-term: ConversationMemory​

2. Long-term: MemoryService (pgvector)​

MCP Server & Tool Registry​

Lifecycle (server_lifespan)​

Tool Registration​

MCP Client — Direct Mode​

Dynamic Tool Selection (get_relevant_tool_schemas)​

LLM Provider — Ollama​

Key Methods​

Thinking Model Support​

Schema Sanitization​

MCP Tools Reference​

Raster Tools (raster_tools.py)​

Analytics Tools (analytics_tools.py)​

Vector Tools (vector_tools.py)​

Knowledge Tools (knowledge_tools.py)​

Memory Tools (memory_tools.py)​

Discovery Tool (discovery.py)​

MCP Resources & Prompts​

Resources​

Prompts​

Data Flow — Step by Step​

Key Design Decisions​

1. In-Process MCP (Direct Mode)​

2. Dynamic Tool Selection​

3. Streaming-First Architecture​

4. UI-Controllable Tools​

5. Two-Level Memory​

6. Concurrency Control via Redis Sorted Set​

Table of Contents