GoogleCloudPlatform / GoogleCloudPlatform/knowledge-catalog
feat(provenance]: an optional `excerpt` field on a `sources` entry
- Dominant language
- TypeScript
- Stars
- 9.2k
- Forks
- 782
- Avg merge
- 6h 36m
- Merged PRs (30d)
- 85
Description
Related to #437
OKF v2, [5.1 provenance sources](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md#51-provenance-sources) attributes a claim to a source by a "_footnote label_" as "_the join key into sources_" entry. The entry has a pointer (`resource`) and three credibility signals (`author`, `usage_count`, and `last_modified`).
```yaml
sources:
- id: ga4-schema
resource: https://developers.google.com/analytics/bigquery/export-schema
title: GA4 BigQuery Export schema
author: team:ga4-docs
usage_count: 5000
last_modified: 2026-05-30T00:00:00Z
usage_window: { from: 2026-06-01T00:00:00Z, to: 2026-06-30T00:00:00Z }
```
**Issue**
OKF v2 does not record the passage (i.e. text extract) the concept relies on.
- A reader (verifier, auditor, detective, reader, human or otherwise) cannot re-check the claim against `resource` without re-reading the entire source and (worst case) having to guess what the author meant.
- `last_modified` says the source changed but can't say the passage, the claim claims <|:-)) still exists.
- Section 5.1 - The footnote can't carry the passage - its not designed for that purpose - "_consumers resolve attribution through the matching entry, not by parsing the footnote prose_".
- Section 5.2 - `verified` records that an actor "_confirmed the content against its sources or resource_" but its not repeatable.
Producer could add a new key to capture, but it has unknown meaning in OKF v2 - so we could improve provenance ...
**Proposal**
An optional field on a `sources` entry, in Section 12 (versioning):
```yaml
sources:
- id: fpa-handbook
resource: https://wiki.example.com/finance/fpa-handbook
title: FP&A reporting handbook
excerpt: "Headline figures are reported for the fiscal year, not the calendar year."
```
- `excerpt`: Its optional. A verbatim copy of the passage in `resource` that the concept relies on, taken as retrieved. Meaningful only when `resource` names a concrete artifact ([section 6.2 - Path valued fields](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/62432a0/okf/SPEC.md#62-path-valued-fields))
- A consumer MAY re-check an excerpt by retrieving `resource` and looking for the passage. OKF stores the excerpt, not the outcome, in the spirit of **Credidibilty Signals** ("_signals, not a verdict_")
- Comparison SHOULD collapse whitespace and MAY ignore case and punctuation. An ellipsis (`...`) inside an excerpt marks omitted text; each part is matched on its own.
- One entry quotes one passage. A source quoted for several claims gets one entry per passage, each with its own `id`, so each footnote still resolves to exactly one excerpt. (NOTE: A list-valued `excerpt` would also work, at the cost of a second list/scalar rule beside the one `verified` already needs)
With #437 (`version`), a verification can then say what was found, by revision, when, by whom. So a refresh can discriminate between "the source moved" to "the passage the claim stood on is gone".
## references
- Linkml pairs a reference field with a `supporting_text` field and checks by normalized substring match [here](https://github.com/linkml/linkml-reference-validator#2-validate-data---full-data-file-validation):
```yaml
# example
Evidence:
attributes:
reference:
range: Reference
implements:
- linkml:authoritative_reference # Marks this as a reference field
supporting_text:
range: string
implements:
- linkml:excerpt # Marks this as text to validate
```
- OKF #160 proposes a `evidence[].quote` inside a larger maturity model.
- W3C Recommendation [Web annotation datamodel](https://www.w3.org/TR/annotation-model/#text-quote-selector) defines a "Text Quote Selector" as "_a range of text by copying it, and including some of the text immediately before (a prefix) and after (a suffix) it to distinguish between multiple copies_"
Contributor guide
Research direction
Start with okf/SPEC.md sections 5.1 and 12, then review the cited LinkML reference validator and related OKF #160 proposal. Define the scope and validation expectations for an optional sources excerpt, including matching and omitted-text behavior; done means the specification clearly records the field and its consumer semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- documentation
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100