[Epic] BRC Assistant: prototype → production (beta launch & beyond)

The BRC Assistant (in-app chat: explore catalog → configure a workflow → hand off to the stepper/Galaxy) has been running behind a feature flag in prototype mode. This epic tracks turning it on for all production users behind a **`beta` tag**, then iterating from real usage. The `beta` tag sets expectations and lets us ship sooner by deferring non-blocking work; observability lands first so we can see the prompts users enter and the responses we return.

Sub-issues are created and linked in GitHub under the phases below.

## Assistant evaluation rubric

A shared, mostly-objective way to score what the assistant does. Two parts: **what** we test (operation × entity), and **how** each query is scored (graded dimensions + the beta bar).

### What we evaluate

Behavior is an **operation × entity matrix**, not a single ladder. Operations order by complexity; entities (**organism / assembly / workflow**) vary in how well each operation is supported.

Per-entity operations (ascending complexity):

1. **Exact lookup** — the user names a thing precisely (exact name or id) and wants its record. *"Tell me about assembly GCF_000005845.2"*
2. **Resolution** — the user names a thing *imprecisely* (nickname, abbreviation, misspelling, outdated name); the assistant must map it to the right catalog entry before it can look anything up. *"Do you have Candida auris?"* — the catalog holds it under its reclassified name, Candidozyma auris.
3. **Filtered list** — the user wants the subset of records matching one or more attributes. *"Which complete-genome assemblies do you have for Cryptococcus neoformans?"*
4. **Subtree / clade** — the user asks about a whole branch of the taxonomy (a genus, family, …); the answer spans every species beneath that node, not one. *"What assemblies do you have for Anopheles?"* → 69 assemblies across 28 species.
5. **Quantitative / aggregate** — the user wants a computed number or grouping over many records (count, max, group-by), not a list. *"How many chromosome-level assemblies are there?"* → 610.

Relational / global (cross-entity):

6. **Compatibility** — can this workflow actually run on this genome? Matches a workflow's constraints (taxonomy, ploidy) against a chosen assembly/organism. *"What workflows can I run on E. coli?"*
7. **Cross-store** — questions that join the genome catalog to a *separate* data store, the SRA/ENA sequencing-reads database (e.g. how much sequencing data exists for an organism). *"How much RNA-seq data exists for Candida auris?"*
8. **Out-of-scope refusal** — the user asks for something the assistant isn't meant to do; it should decline cleanly and redirect, not fake an answer. *"What's the weather in Paris?"* / *"Search all of ENA for X."*

### How each query is scored

| Dimension | 🟢 | 🟡 | 🔴 |
|---|---|---|---|
| **Correctness** | works (supported + tested) | degrades (honestly "can't yet") | blocker (silently wrong / plausible-but-wrong) |
| **↳ Grounding** *(within Correctness)* | every factual/evaluative claim traces to a catalog field | minor general-knowledge aside, clearly framed as such | asserts unverifiable facts or recommendations the catalog didn't make ("gold-standard", "best choice") |
| **Responsiveness** | answers directly, answer-first; addresses the intent | answers, but buried / the user must derive it | doesn't answer, or answers a different question |
| **Latency** (wall-clock) | ≤ 8s | 8–15s | > 15s |
| **Cost** (USD/turn) | ≤ $0.03 | $0.03–$0.08 | > $0.08 |

**Style:** tone, verbosity, exuberance, formatting. Decompose into named sub-criteria; proxy verbosity with **output-token count**; judge tone against a written style guide, scored for consistency.

**Beta bar:** the bar is that a response **honestly degrades** ("can't do that yet") rather than being **silently wrong** — and a response can be silently wrong in more than one way (a wrong answer, a correct answer to the *wrong* question, or a confident *ungrounded* claim). Which cells must be "works" versus may "degrade" for launch is a team/triage decision, not fixed by this rubric.

**Cost basis:** `claude-sonnet-4-6` list price, $3/1M input + $15/1M output; `cost = in×$3/1M + out×$15/1M`.

**Diagnostics recorded (not graded):** `requests`, `tool_calls`, tokens in/out. Invariant: `requests = tool_calls + 1`.

### Scorecard — worked examples (live `claude-sonnet-4-6`, real catalog)

Real queries scored against the rubric above — concrete examples of how each dimension lands in practice (one row per failure mode).

| Query type (with example) | correct | grounding | responsive | latency | cost |
|---|---|---|---|---|---|
| Exact lookup — *"Do you have Trypanosoma brucei?"* | ✅ | ✅ | ✅ | 6.7s | $0.025 |
| Exact lookup — *"Tell me about assembly GCF_000005845.2"* | ✅ | 🟡 adds "most well-characterized" (not in catalog) | ✅ | ~7s | ~$0.03 |
| Filtered list, 82 rows — *"complete-genome assemblies for C. neoformans?"* | 🔴 undercounts at scale | ✅ | ✅ | 25.7s | $0.118 |
| Filtered list, 108 rows — *"complete-genome assemblies for A. fumigatus?"* | 🔴 silently broadens the filter | ✅ | ✅ | 24.3s | $0.139 |
| Yes/no lookup — *"Do we have a complete-genome reference for C. albicans?"* | ✅ | ✅ | 🔴 never answers the yes/no | ~14s | ~$0.10 |
| Subtree / clade — *"What assemblies do you have for Anopheles?"* | 🔴 partial — 11 of 28 species, 28 of 69 assemblies, shown as complete | ✅ | ✅ | 13.0s | $0.048 |
| Aggregate — *"How many chromosome-level assemblies are there?"* | 🟡 degrades — honestly can't count catalog-wide (real answer 610) | ✅ | ✅ | 8.6s | $0.014 |

**Key property:** Correctness / Latency / Cost move together on the corpus-dump cases — one fix (tool-side filtering returning `{total, capped rows, facets}`) green-lights all three. Responsiveness and Grounding are independent failure modes that surface even when the data is correct (the C. albicans and "gold-standard" rows).

## Evaluation list

The concrete prompts that exercise the matrix, each with what it checks. Diverse taxa on purpose (a query that passes on a 27-assembly organism can fail on a 108-assembly one).

Each eval is scored against all rubric dimensions — correctness (incl. grounding), responsiveness, latency, and cost — captured automatically by the probe (`time=…s cost=$…`). The status flags below call out known correctness gaps only (as of 2026-06-15).

### Operation evaluations

**1. Exact lookup**
- `org-by-name` — *"Do you have Plasmodium falciparum?"* — a precise species name returns its catalog record with the taxid echoed back.
- `org-by-id` — *"What organism has NCBI taxonomy id 5833?"* — a reverse id→name lookup resolves and is grounded in the catalog.
- `org-miss` — *"Do you have Plasmodium falcipar22m?"* — a non-existent name degrades to "did you mean…", never a silent empty result.
- `asm-by-accession` — *"Tell me about assembly GCF_000005845.2."* — a precise accession returns full assembly facts.
- `asm-miss` — *"Tell me about assembly GCF_999999999.9."* — an unknown accession returns an honest not-found plus a redirect.
- `wf-categories` — *"What kinds of analyses can I run?"* — the analysis categories are listed with workflow counts.
- `wf-in-category` — *"What transcriptomics workflows do you have?"* — the contents of a named category are listed.

**2. Resolution**
- `res-common-name` — *"Do you have yeast?"* — a colloquial name resolves to the right species.
- `res-abbreviation` — *"Tell me about T. cruzi."* — a genus-abbreviated binomial expands to the correct species.
- `res-from-description` — *"Do you have the parasite that causes Chagas disease?"* — the assistant identifies the organism from a description and grounds it (Trypanosoma cruzi).
- `res-less-common` — *"Do you have Schistosoma mansoni?"* — resolution works for taxa well outside the popular set.
- `res-real-but-wrong` — *"I want to study P. vivax, the deadliest human malaria parasite."* — the assistant honors the named taxon and flags the name/description conflict instead of silently substituting.
- `res-synonym` — *"Do you have Candida auris?"* — a reclassified name resolves to its current taxon (Candidozyma auris) rather than a false "not in catalog". **(blocker today)**
- `res-versionless-accession` — *"Tell me about assembly GCF_000005845."* — a version-stripped accession resolves to the unique versioned record. **(inconsistent today)**

**3. Filtered list**
- `filter-level` — *"Which complete-genome assemblies do you have for Cryptococcus neoformans?"* — filtering by assembly level returns the correct subset. **(wrong at scale today)**
- `filter-reference` — *"Show me only the reference assembly for Plasmodium vivax."* — the reference-only (`isRef`) filter returns just the reference.
- `filter-vague` — *"Show me the good-quality assemblies for Toxoplasma gondii."* — the assistant discloses how it interpreted an undefined adjective rather than silently choosing.
- `filter-compound-empty` — *"Reference assemblies that are complete-genome level for Candida albicans?"* — an empty intersection is reported as "none", not by quietly dropping a constraint. **(fails today)**
- `filter-catalog-wide` — *"Which complete-genome assemblies do you have?"* (no organism) — degrades gracefully when there's no catalog-wide filter tool, rather than fabricating.
- `filter-wf-gtf` — *"Which workflows require a gene annotation (GTF)?"* — filtering workflows by a requirement returns the right set. **(brute-force/expensive today)**

**4. Subtree / clade**
- `clade-assemblies` — *"What assemblies do you have for Anopheles?"* — a genus query returns all assemblies across descendant species (69 across 28), not the ~0 from exact-match. **(blocker; not yet run live)**
- `clade-organisms` — *"What organisms do you have under Apicomplexa?"* — a higher-rank query lists the member species beneath it.

**5. Quantitative / aggregate**
- `agg-count` — *"How many chromosome-level assemblies are there?"* — the count is computed (610), not eyeballed from a list. **(expected fail today)**
- `agg-group-by` — *"How many assemblies are there per assembly level?"* — a grouped count returns a number per category.

**6. Compatibility**
- `compat-organism` — *"What workflows can I run on E. coli?"* — lineage-aware matching includes clade-scoped workflows (e.g. Bacteria-level), not just exact-taxid ones.
- `compat-check` — *"Is the RNA-seq workflow compatible with GCF_000146045.2?"* — a specific workflow×assembly check validates ploidy / taxonomy / GTF.
- `compat-coming-soon` — *"Do you have comparative genomics workflows?"* — the assistant says "coming soon / can't drive it here" knowingly, rather than falsely claiming none exist.

**8. Out-of-scope refusal**
- `refuse-offtopic` — *"What's the weather in Paris?"* — a clean decline plus a redirect to what the assistant is for.
- `refuse-capability` — *"Search all of ENA for X."* — it declines a search it can't do and points to the ENA picker.

*(Operation 7, cross-store, is excluded — behind the `SRA_MIRROR_PATH` flag / draft PR #1302.)*

### Cross-cutting checks (apply to any query)

- **Responsiveness** — the response answers the literal question, answer-first, even when the answer is "no". *Example: "Do we have any complete-genome reference for C. albicans?"*
- **Grounding** — the reply asserts only catalog-backed facts, with no ungrounded superlatives ("gold-standard", "best choice"). *Example: "Tell me about assembly GCF_000005845.2."*
---

## Design principles

The architectural rules behind the fixes — what "good" looks like, independent of any single ticket.

1. **Summaries, never the corpus.** The model emits a query and reads a summary — `{total, capped rows, facets}` — never the raw row set. *Never enumerate to count*: "how many" compiles to a count, not list-then-tally.
2. **Same contract at every scale.** The tool's input/output shape is fixed; only the executor swaps (in-memory scan today → DuckDB/Solr aggregation later). Scaling the backend never changes the agent contract.
3. **Entity resolution: the LLM normalizes, exact lookup grounds.** The model emits a *name*, never an id; a deterministic table maps name→taxid. Curated aliases + NCBI synonyms are *exact keys*, not fuzzy retrieval (trigram is a miss-path fallback; embeddings are discovery-only, since the taxon name space is bounded).
4. **Echo the resolved entity (name + id) back.** Exact lookup catches not-in-catalog misses structurally; the confident *wrong-but-real* resolution is caught only by showing the user what we resolved to.
5. **Subtree via materialized lineage.** Use the stored ancestor lineage (`lineageTaxonomyIds`) as an ancestor-membership / semi-join, not a recursive walk. Carry a `subtree` flag (exact vs. descendants).
6. **Agent as the spine, one typed query tool for analytics.** Borrowing from compilers, an **intermediate representation (IR)** is a structured, typed stand-in for a request that sits *between* the natural-language question and the query that actually runs — just as a compiler turns source code into an intermediate form before emitting machine code. The model never writes a raw database query; it fills a **typed tool-argument schema** (that schema *is* the IR), and the backend **compiles it deterministically** into the real query (an in-memory scan today, DuckDB/SQL/Solr later). Reach for this shape when a query needs joins, nesting, or pre-execution inspection (count, aggregate, subtree, "which X have Y").
7. **Grounding: every claim traces to a catalog field.** No ungrounded facts or recommendations ("gold-standard", "best choice"); catalog-backed signals only (`isRef`, annotation, level).
8. **Honest degrade over silent-wrong.** Tools surface `total`/`truncated` so the model can say "showing 10 of N" or "I can't do that yet" rather than presenting a partial or eyeballed answer as complete.
9. **State changes flow through typed tool calls, not parsed from the reply** *(proposed)*. The assistant produces two distinct outputs: the human-readable message, and machine state (workflow-config fields, suggestion chips, handoff). State mutations should be emitted as dedicated, validated tool calls the harness intercepts — typed, schema-validated, gate-able, and decoupled from how the model phrases its prose — rather than parsed out of the conversational output.

---

## Phase 1 — Enter beta (remove feature flag)

Remove the feature flag and enable the assistant for all production users behind a `beta` tag. The show-stoppers that gate entering beta.

**TODO (triage):** decide which of these are true show-stoppers for entering beta, which can move to Phase 2, and which can be dropped or modified.

**Open**

- #1290
- #1293
- #1299
- #1318
- #1320

**Complete**

- #1289
- #1291
- #1292
- #1294
- #1295
- #1296
- #1297
- #1316
- #1319
- #1321
- #1354
- #1357
- #1371 — organism search on the query IR + DuckDB; don't silently truncate (uses current organisms.json — no #1367/#1368 dependency) [go-live blocker]

## Phase 2 — Exit beta (remove beta tag)

Blockers that must be resolved to exit beta and remove the `beta` tag.

- #1298

## Phase 3 — Post-beta work

New features and integrations after the `beta` tag is removed.

- #1300
- #1369 — multi-turn context-dependence eval set (gate for assistant history-thinning)
- #1370 — thin replayed history to cut tokens (gated on #1369)
- #1372 — resolve organism names via catalog synonyms (depends on #1368, #1371)