Run conformance against the upstream OKF sample bundles
- Dominant language
- Dart
- Stars
- 0
- Forks
- 0
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 31
Description
## Summary
`test/conformance_test.dart` covers synthetic OKF 0.2 §11 cases built in memory against the pinned revision `3fcbb9f`. It never loads the bundles upstream ships, so it could not detect spec drift such as [knowledge-catalog#323](https://github.com/GoogleCloudPlatform/knowledge-catalog/pull/323) or real-bundle cases the synthetic tests miss.
Current `okf` 0.3.0 results (through Wayfinder 0.0.3) against `knowledge-catalog` at `a430739`:
| Upstream bundle | Result | Findings |
| --- | --- | --- |
| `okf/bundles/ga4` | PASS | none |
| `okf/bundles/stackoverflow` | PASS | none |
| `okf/bundles/crypto_bitcoin` | PASS | none |
| `okf/bundles/acme_retail` | FAIL | error `okf/invalid-log-frontmatter` (its `log.md` has YAML frontmatter), plus `invalid-stale-after` and `invalid-usage-window` advisories caused by the timestamp change |
The `okf/samples/*` directories fail only because their `README.md` files have no frontmatter. They look like source material rather than bundles.
## Proposal
- Fetch or vendor upstream `okf/bundles` at a pinned commit and assert an expected result per bundle: the verdict and finding IDs. That gives an upgrade gate.
- Add a scheduled job against upstream `main` that reports drift without failing releases.
- Exclude `okf/samples`, or record why it isn't a bundle.
- The `acme_retail` `log.md` case depends on [knowledge-catalog#286](https://github.com/GoogleCloudPlatform/knowledge-catalog/issues/286), which asks whether `log.md` may have frontmatter. Record it as a known difference until the spec decides.
Related: #56 (the timestamp spec change) and #57 (strict timestamp parsing).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with test/conformance_test.dart, then inspect the upstream okf/bundles paths and pinned revisions described in the issue. Run the existing conformance tests and compare bundle verdicts and finding IDs. Done means pinned bundles have expected results, okf/samples is excluded or documented, acme_retail is recorded as a known difference, and scheduled upstream-main drift reporting is defined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- dart
- Domain
- ci-cd, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100