GoogleCloudPlatform / GoogleCloudPlatform/knowledge-catalog

Proposal: optional `retrieved_at` / `verified_at` and a source-liveness convention — the container outlives the source

Open
#211 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
9.2k
Forks
782
Avg merge
6h 36m
Merged PRs (30d)
85

Description

## Summary

§4.1 gives a concept one clock: `timestamp`, "last meaningful change." For bundles whose concepts point at external `resource` URLs, that conflates two things a consumer needs to distinguish — when the *content* changed vs. when a producer last *fetched or verified* it — and the spec has no home at all for a third signal: whether the `resource` still resolves.

This is adjacent to #97 (elevate `timestamp`) and the maintenance-signals discussion in #158, but scoped to the producer-side fields those threads don't cover: retrieval/verification dates and source liveness.

## Evidence (civic public records, where sources rot fast)

We maintain an OKF v0.1 bundle of local-government records for a large county in the US Northeast. Within the bundle's first year we hit, in production:

- **Sources vanishing under live atoms:** a town's public-hearing notices disappeared from its news feed before the year ended. The atom is fine; its `resource` is gone. Nothing in the format records or reveals that.
- **Verification as a distinct event:** libraries post board minutes months late; a fact can be *re-verified unchanged* long after its content `timestamp`. A consumer deciding whether to trust a figure needs "verified last week" vs. "verified last year," which `timestamp` cannot express without destroying its content-change meaning.
- **Liveness is checkable but nontrivial:** we run a sweep that GETs every `resource` in the bundle (~160 URLs). Getting truthful answers required a browser user-agent, per-host politeness delays, and 429 backoff — civic-portal hosts throttle and UA-block. It's exactly the kind of shared problem a documented convention (plus a reference tool) would keep every producer from re-solving.

## Proposal (all optional, all additive under §4.1 / §11)

1. `retrieved_at` — when the producer last fetched the `resource`.
2. `verified_at` — when a human or pipeline last confirmed the concept still matches its source.
3. A `source_status` convention — last-checked date + last observed HTTP state for the `resource` (e.g. `{checked: 2026-07-02, state: ok | dead | blocked}`).

Consumers that don't know the fields ignore them; nothing about §9's permissive-consumption contract changes. A v0.1 bundle stays conformant.

We have a working liveness checker (Python, no dependencies beyond the standard library + requests) built for this bundle and would be glad to contribute it as a reference implementation if there's interest.

Contributor guide

Open the contributing guide

Research direction

Start with §4.1, §9, and §11, then compare adjacent issues #97 and #158. Evaluate the three optional producer-side signals and the proposed liveness convention, including the offered Python checker; done means the project reaches a documented decision on scope and compatibility.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.