Architecture map: UK Chat pipeline, methodology, and implementation implications

Purpose

Provide an at-a-glance map of what sits behind UK Chat: the end-to-end request pipeline, the calculation method, the deterministic/non-deterministic boundary, and the main implications of the current design.

This is a snapshot of the implementation as of 29 July 2026, based on the active code and the engineering/runtime documentation.

1. End-to-end pipeline

flowchart LR
    U[User] --> UI[Next.js chat UI]
    UI --> PX[Next.js backend proxy]
    PX --> API[FastAPI /chat/message]

    API --> GUARD[Auth/balance checks + rate limits]
    GUARD --> FOLLOW{Opening turn?}
    FOLLOW -->|Yes| GW[Gateway: forced emit_plan call]
    FOLLOW -->|No| MS[Model selection]

    GW --> GATE{Deterministic gate}
    GATE -->|needs_plan / partial / out_of_scope / irrelevant| LW[Lightweight writer<br/>no tools supplied]
    GATE -->|ready| MS

    MS -->|simple / smaller context| FAST[Fast model]
    MS -->|large context| COMPLEX[Complex model]
    MS -->|reform / distributional / charts| REASON[Reasoning model]

    FAST --> LOOP[Anthropic streaming tool loop]
    COMPLEX --> LOOP
    REASON --> LOOP

    LOOP --> DISC[Discovery]
    DISC --> VAL[Validation]
    VAL --> SIM[Household or society simulation]
    SIM --> DERIV[Official derivative adapters]
    DERIV --> CHART[Deterministic chart spec]
    CHART --> LOOP
    LOOP --> SSE[SSE: text + tool events + done]
    LW --> SSE

    SSE --> UI
    SSE --> OPS[Usage billing + traces/metrics]
    UI --> STORE[Conversation persistence/share/report]

Key runtime detail: the gateway runs only on the opening turn. Follow-ups go directly to model selection because they depend on conversation context and may be answers to a clarification request.

2. Society calculation and trust boundary

flowchart TB
    subgraph Nondeterministic[Model-mediated]
        INTENT[Interpret user intent]
        PLAN[Choose tools and arguments]
        PROSE[Write the explanation]
    end

    subgraph Deterministic[Application-enforced]
        SCHEMA[Typed schemas + request validation]
        DISPATCH[Registered-tool dispatch]
        CTX[Turn-local opaque result store]
        PE[policyengine.py + policyengine-uk]
        OUT[Official weighted output classes]
        SPEC[Fixed chart transformation]
    end

    INTENT --> PLAN --> SCHEMA --> DISPATCH
    DISPATCH --> PE
    PE -->|baseline + reform Simulation objects| CTX
    CTX -->|simulation_id only| DISPATCH
    DISPATCH --> OUT
    OUT -->|typed derivative result + result_id| CTX
    CTX --> SPEC
    SPEC --> PROSE

    RAW[(Enhanced FRS 2024-25<br/>release 1.56.13)] --> PE
    RAW -. raw rows never returned .-> BLOCK[No row-level tool result]

For society-wide analysis, UK Chat pins enhanced_frs_2024_25 at release 1.56.13. run_society_simulation creates baseline/reform simulations in memory and returns an opaque, turn-local handle. Derivative tools retrieve that handle and delegate weighted calculations to official policyengine.py output classes. Raw survey rows are not exposed to the model, and pandas tabular objects are rejected by serialization.

3. Responsibility map

Layer	Owns	Method / boundary
Frontend	Chat UX, streaming display, working/tool panels, chart rendering, conversations, sharing/reporting	Consumes SSE and renders backend-produced chart specs
API	FastAPI assembly, CORS, rate limits, safe JSON, health/version	HTTP and operational boundary
Gateway	Opening-turn grounding and routing	Fast model grounds slots; pure server policy decides one of five outcomes
Chat orchestrator	Model selection, prompts, streaming, parallel tool execution, iteration control	Direct Anthropic SDK loop; max 30 tool iterations
Tool seam	Discovery → validation → simulation → derivatives → artifacts	Only @register_tool functions are model-facing; no arbitrary Python tool
Engine	UK model loading, reforms, households, simulations, official derivatives	policyengine.py + policyengine-uk; default simulation year 2026
State	Turn-local result handles; persisted conversation metadata/messages	Simulation objects stay in-process and do not cross the tool boundary
Operations	Billing, usage, tracing, metrics, errors	Records gateway/model/tool phases and bills actual model usage
Quality	Unit/integration tests plus offline/live evals	Tests enforce runtime contracts; evals cover routing, tool use, answers, microdata safety, and trajectories

4. What is deterministic vs model-mediated?

Deterministic / enforced in code	Model-mediated / probabilistic
Request/schema validation	Understanding the user’s wording
Gateway criticality and final gate outcome	Gateway slot grounding
No-tools structure on lightweight routes	Tool selection and argument planning on compute routes
Registered-tool dispatch	Whether/when to call the available tools
Typed simulation execution and official derivative calculations	Final prose and explanation structure
Fixed dataset and default year	Follow-up suggestions (intentionally sampled)
Result truncation, chart JSON construction, billing and persistence	Conversation title generation

The compute, gateway, title, and eval model calls use temperature 0; suggestion chips use a separate sampling temperature. Temperature 0 improves repeatability but does not turn model planning into an application-level guarantee.

5. Main implementation implications

Benefits

• Calculation integrity: quantitative answers flow through discoverable, typed tools rather than recalled values or model-authored code.
• Privacy: society microdata remains behind the engine boundary; the model receives summaries and typed derivatives, not survey rows.
• Auditability: the server owns gate policy, schemas, dispatch, dataset choice, derivative methods, and chart transforms.
• Efficiency: irrelevant or under-specified opening turns can use a cheap no-tools route; independent tool calls within an iteration execute concurrently.
• Extensibility: registering one tool populates the canonical definition and handler surfaces, reducing schema/dispatcher drift.

Trade-offs and watchpoints

• The model still plans the compute path. Schemas and prompts guide tool choice, but only deliberately forced routing would make a particular tool call deterministic.
• Gateway context is opening-turn only. This avoids misclassifying contextual replies, but follow-ups rely entirely on the compute prompt/transcript rather than a refreshed structured plan.
• Fail-safe favours availability and capability over cost. Gateway errors route to the full compute loop rather than refusing or dropping a turn.
• Handles are turn-local. This protects internal objects and simplifies privacy, but simulations cannot be reused across requests without a deliberate persistence design.
• The dataset is an application invariant. Reproducibility is strong, but updating Enhanced FRS requires a code change and redeployment.
• The tool loop is bounded. A 30-iteration cap protects the SSE/proxy path from runaway agents, but complex requests can terminate with an iteration-cap fallback.
• Aggregate correctness depends on upstream official outputs. Runtime adapters intentionally avoid custom local weighting; changes to output semantics should therefore be reviewed at the policyengine.py boundary.
• Conversation, billing, and simulation state have different lifetimes. Conversations/usage persist, while calculation objects remain in the turn process; incident debugging should correlate them through session IDs and observability rather than expect persisted simulation objects.

6. Source-of-truth entry points

• docs/engineering/skills/uk-chat-runtime.md
• backend/chat/orchestrator.py
• backend/gateway/runtime.py and backend/gateway/policy.py
• backend/tools/dispatch.py and backend/tools/context.py
• backend/engine/
• docs-site/architecture.md

Review request

@anth-volk — could you take a look and share your opinion, especially on whether this captures the implementation accurately and whether the trade-offs/watchpoints are the right ones to highlight? I’d also value your view on whether this should become a maintained architecture page (or diagram) in the docs, and what you would change before we do that.

Read docs/engineering/skills/uk-chat-runtime.md and docs-site/architecture.md, then compare the architecture map with the source-of-truth files in backend/chat, backend/gateway, backend/tools, and backend/engine. Check whether the pipeline, boundaries, and trade-offs accurately reflect the implementation, and determine what should change before this becomes a maintained architecture page or diagram.