GoogleCloudPlatform / GoogleCloudPlatform/knowledge-catalog
Concepts are addressable by topic, but retrieval failures are shaped like symptoms
- Dominant language
- TypeScript
- Stars
- 9.2k
- Forks
- 782
- Avg merge
- 6h 36m
- Merged PRs (30d)
- 85
Description
## The failure mode
A concept can be present, correct, well-titled and still never load, because the
consumer never forms the query that would reach it. This is not a coverage
problem and not a freshness problem — the two the spec already has signals for.
The corpus was right; it just wasn't addressable from where the consumer stood.
We hit this repeatedly enough to measure it, and I think it is worth a field
report because the shape is consistent.
## Field data
Bundle: 148 concepts, ~950 KB, in daily use since June 2026 by an LLM coding
agent working across five repos. Conversion to OKF cut per-question context 59%
(7,164 → 2,937 tokens average over 12 real questions), so retrieval works well
*once aimed*.
In a single working session the agent acted on four wrong assumptions. In **all
four**, the correct answer was already in the bundle:
| What the agent concluded | Reality | Where it was documented |
|---|---|---|
| "Docker isn't installed on this machine, integration tests can't run here" | Installed; daemon stopped | 2 concepts — one written *because the same mistake happened 3 weeks earlier* |
| Pushed straight to `main` | Repo is branch+PR only | 1 concept + the repo's own agent instructions |
| Debugged a Graph API `400` as an auth problem | Documented gotcha, wrong layer | 1 concept, 3 days old |
| Started authoring a new concept | Near-duplicate already existed | the section index |
The fourth is the control case: it is the only one where the agent searched
**before** acting rather than after, and it is the only one that was caught.
So the hit rate of the corpus, conditional on being consulted, was 4/4. The
problem is entirely in what causes consultation.
## Why the index didn't fire
Our index lines are curated one-liners. The Docker one reads:
```
* [Docker on this Windows](reference_docker_windows_testcontainers.md) - it IS here; without it the *_integ.py ERROR, they don't skip
```
That is a good line for someone already browsing that section. But what the
agent actually had in front of it was:
```
open //./pipe/dockerDesktopLinuxEngine: The system cannot find the file specified
```
There is no overlap. Not a single token of the observed evidence appears in the
addressing surface. The same is true of the Graph API case: the concept was
titled `WhatsApp templates in Meta (bookings + schedules)` and the agent was
working on an *alert* template — the parenthetical actively signalled "not
yours" while the body contained the exact answer.
**Concepts are indexed by what they are about. Failures arrive as a symptom: an
error string, a command about to run, a verb.** Topic-addressing requires the
consumer to already suspect the topic, which is precisely what is missing when
the consumer is confident and wrong. Confidence suppresses retrieval, and
confidence is uncorrelated with correctness.
## Relationship to #254
#254 (telemetry on corpus usage) reaches the edge of this from the other side:
> A concept that is never read might be redundant, **might be badly titled,
> might be described in terms no consumer queries for**, or might be the one
> thing pulled once a year for an audit. Usage data does not resolve which.
Agreed, and this issue is the complement. #254 asks how to *observe* that a
concept is cold. This asks what makes a concept *reachable* in the first place.
A cold-concept metric plus a symptom-addressing surface would together let you
answer "cold because nobody needed it" vs "cold because nobody could name it".
## What we are trying
Since §"Producers MAY include any additional keys" makes this legal without a
spec change, we are adding an optional producer key carrying the literal strings
that should reach the concept — error text, commands, action verbs:
```yaml
triggers:
- "cannot find the file specified"
- "docker info"
- "integration tests can't run here"
```
and emitting them **inline in the always-loaded root index**, not behind a link.
A link does not fire; only text already in the window fires. For our bundle only
~16 of 148 concepts warrant this, which is the interesting part: the ones worth
preloading are the ones whose absence fails **silently**.
That distinction seems generalizable. Missing "how to deploy the API" is
self-correcting — the consumer knows it doesn't know and goes looking. Missing
"the thing you are about to conclude is impossible is actually possible" is not:
the consumer proceeds confidently and ships unverified work, and nobody finds
out. Cost of a miss is asymmetric, so preloading budget should follow that
asymmetry rather than frequency.
Cost for us: ~840 tokens once per session, against a 244k-token corpus.
## The part we do not have a good answer for
**Triggers are authored by the person who already understands the concept**, and
that person is the worst possible guesser of which symptom a confused consumer
will present with. Written at authoring time they will drift toward restating
the title.
Our current mitigation is to harvest them *post-incident* — the trigger string
is the error actually observed at the moment of failure, copied verbatim. That
makes the field a record of real misses rather than a prediction of hypothetical
ones, but it also means the first miss is never prevented, only the repeats. For
us that would still have caught 2 of the 4 above, since both were repeats.
Whether that is worth standardizing, or whether it belongs entirely in producer
space, is the open question. I am not proposing a required field — mostly I want
to know if others are seeing the same shape, and whether anyone has found a
mechanism that fires on evidence rather than on the consumer's self-assessment.
Happy to share the bundle's measurement tooling (context measurement, retrieval
tests, broken-link audit) if useful — it is how we got the numbers above.
Contributor guide
Research direction
Start by reading the proposed producer `triggers` key, the always-loaded root index, and the relationship to issue #254; the Docker example is in `reference_docker_windows_testcontainers.md`. The issue does not identify implementation files or tests, and “done” remains unresolved between standardizing the field and leaving it to producers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript, yaml
- Domain
- documentation, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100