GoogleCloudPlatform / GoogleCloudPlatform/knowledge-catalog

Concepts are addressable by topic, but retrieval failures are shaped like symptoms

Open
#265 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
9.2k
Forks
782
Avg merge
6h 36m
Merged PRs (30d)
85

Description

## The failure mode

A concept can be present, correct, well-titled and still never load, because the
consumer never forms the query that would reach it. This is not a coverage
problem and not a freshness problem — the two the spec already has signals for.
The corpus was right; it just wasn't addressable from where the consumer stood.

We hit this repeatedly enough to measure it, and I think it is worth a field
report because the shape is consistent.

## Field data

Bundle: 148 concepts, ~950 KB, in daily use since June 2026 by an LLM coding
agent working across five repos. Conversion to OKF cut per-question context 59%
(7,164 → 2,937 tokens average over 12 real questions), so retrieval works well
*once aimed*.

In a single working session the agent acted on four wrong assumptions. In **all
four**, the correct answer was already in the bundle:

| What the agent concluded | Reality | Where it was documented |
|---|---|---|
| "Docker isn't installed on this machine, integration tests can't run here" | Installed; daemon stopped | 2 concepts — one written *because the same mistake happened 3 weeks earlier* |
| Pushed straight to `main` | Repo is branch+PR only | 1 concept + the repo's own agent instructions |
| Debugged a Graph API `400` as an auth problem | Documented gotcha, wrong layer | 1 concept, 3 days old |
| Started authoring a new concept | Near-duplicate already existed | the section index |

The fourth is the control case: it is the only one where the agent searched
**before** acting rather than after, and it is the only one that was caught.

So the hit rate of the corpus, conditional on being consulted, was 4/4. The
problem is entirely in what causes consultation.

## Why the index didn't fire

Our index lines are curated one-liners. The Docker one reads:

```
* [Docker on this Windows](reference_docker_windows_testcontainers.md) - it IS here; without it the *_integ.py ERROR, they don't skip
```

That is a good line for someone already browsing that section. But what the
agent actually had in front of it was:

```
open //./pipe/dockerDesktopLinuxEngine: The system cannot find the file specified
```

There is no overlap. Not a single token of the observed evidence appears in the
addressing surface. The same is true of the Graph API case: the concept was
titled `WhatsApp templates in Meta (bookings + schedules)` and the agent was
working on an *alert* template — the parenthetical actively signalled "not
yours" while the body contained the exact answer.

**Concepts are indexed by what they are about. Failures arrive as a symptom: an
error string, a command about to run, a verb.** Topic-addressing requires the
consumer to already suspect the topic, which is precisely what is missing when
the consumer is confident and wrong. Confidence suppresses retrieval, and
confidence is uncorrelated with correctness.

## Relationship to #254

#254 (telemetry on corpus usage) reaches the edge of this from the other side:

> A concept that is never read might be redundant, **might be badly titled,
> might be described in terms no consumer queries for**, or might be the one
> thing pulled once a year for an audit. Usage data does not resolve which.

Agreed, and this issue is the complement. #254 asks how to *observe* that a
concept is cold. This asks what makes a concept *reachable* in the first place.
A cold-concept metric plus a symptom-addressing surface would together let you
answer "cold because nobody needed it" vs "cold because nobody could name it".

## What we are trying

Since §"Producers MAY include any additional keys" makes this legal without a
spec change, we are adding an optional producer key carrying the literal strings
that should reach the concept — error text, commands, action verbs:

```yaml
triggers:
- "cannot find the file specified"
- "docker info"
- "integration tests can't run here"
```

and emitting them **inline in the always-loaded root index**, not behind a link.
A link does not fire; only text already in the window fires. For our bundle only
~16 of 148 concepts warrant this, which is the interesting part: the ones worth
preloading are the ones whose absence fails **silently**.

That distinction seems generalizable. Missing "how to deploy the API" is
self-correcting — the consumer knows it doesn't know and goes looking. Missing
"the thing you are about to conclude is impossible is actually possible" is not:
the consumer proceeds confidently and ships unverified work, and nobody finds
out. Cost of a miss is asymmetric, so preloading budget should follow that
asymmetry rather than frequency.

Cost for us: ~840 tokens once per session, against a 244k-token corpus.

## The part we do not have a good answer for

**Triggers are authored by the person who already understands the concept**, and
that person is the worst possible guesser of which symptom a confused consumer
will present with. Written at authoring time they will drift toward restating
the title.

Our current mitigation is to harvest them *post-incident* — the trigger string
is the error actually observed at the moment of failure, copied verbatim. That
makes the field a record of real misses rather than a prediction of hypothetical
ones, but it also means the first miss is never prevented, only the repeats. For
us that would still have caught 2 of the 4 above, since both were repeats.

Whether that is worth standardizing, or whether it belongs entirely in producer
space, is the open question. I am not proposing a required field — mostly I want
to know if others are seeing the same shape, and whether anyone has found a
mechanism that fires on evidence rather than on the consumer's self-assessment.

Happy to share the bundle's measurement tooling (context measurement, retrieval
tests, broken-link audit) if useful — it is how we got the numbers above.

Contributor guide

Open the contributing guide

Research direction

Start by reading the proposed producer `triggers` key, the always-loaded root index, and the relationship to issue #254; the Docker example is in `reference_docker_windows_testcontainers.md`. The issue does not identify implementation files or tests, and “done” remains unresolved between standardizing the field and leaving it to producers.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript, yaml
Domain
documentation, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.