conceptadev / conceptadev/okf

Run conformance against the upstream OKF sample bundles

Open
#58 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Dart
Stars
0
Forks
0
Avg merge
1d 1h
Merged PRs (30d)
31

Description

## Summary

`test/conformance_test.dart` covers synthetic OKF 0.2 §11 cases built in memory against the pinned revision `3fcbb9f`. It never loads the bundles upstream ships, so it could not detect spec drift such as [knowledge-catalog#323](https://github.com/GoogleCloudPlatform/knowledge-catalog/pull/323) or real-bundle cases the synthetic tests miss.

Current `okf` 0.3.0 results (through Wayfinder 0.0.3) against `knowledge-catalog` at `a430739`:

| Upstream bundle | Result | Findings |
| --- | --- | --- |
| `okf/bundles/ga4` | PASS | none |
| `okf/bundles/stackoverflow` | PASS | none |
| `okf/bundles/crypto_bitcoin` | PASS | none |
| `okf/bundles/acme_retail` | FAIL | error `okf/invalid-log-frontmatter` (its `log.md` has YAML frontmatter), plus `invalid-stale-after` and `invalid-usage-window` advisories caused by the timestamp change |

The `okf/samples/*` directories fail only because their `README.md` files have no frontmatter. They look like source material rather than bundles.

## Proposal

- Fetch or vendor upstream `okf/bundles` at a pinned commit and assert an expected result per bundle: the verdict and finding IDs. That gives an upgrade gate.
- Add a scheduled job against upstream `main` that reports drift without failing releases.
- Exclude `okf/samples`, or record why it isn't a bundle.
- The `acme_retail` `log.md` case depends on [knowledge-catalog#286](https://github.com/GoogleCloudPlatform/knowledge-catalog/issues/286), which asks whether `log.md` may have frontmatter. Record it as a known difference until the spec decides.

Related: #56 (the timestamp spec change) and #57 (strict timestamp parsing).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with test/conformance_test.dart, then inspect the upstream okf/bundles paths and pinned revisions described in the issue. Run the existing conformance tests and compare bundle verdicts and finding IDs. Done means pinned bundles have expected results, okf/samples is excluded or documented, acme_retail is recorded as a known difference, and scheduled upstream-main drift reporting is defined.

Written by the indexing model from the issue text.

Assessment

Tech stack
dart
Domain
ci-cd, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.