Publishing provider conformance: a machine-readable report, and how to collect one from providers we do not host

Follow-up to #417, which asks in its open questions: "Who owns conformance claims? Is there a badge, a report format, a listing on openfeature.dev? Or is passing simply its own reward for now?" This issue is that question, split out so it can progress on its own timeline.

This is a design discussion, not a proposal to implement.

Status, 2026-09-11

The report format is no longer the substance of this issue. Following @toddbaert's comment below, #425 now defines only a small envelope and carries the results as Cucumber Messages rather than a format of our own. The sections below have been trimmed to what that leaves open: collection, trust and expiry. The minimized comments record how the format got there.

The opportunity

The provider conformance suite (Appendix F) has implementations in four languages, each of which already computes the data a public conformance report needs: the capability set a provider declares, and a per-scenario outcome. What is missing is a way to publish and collect reports. That would let openfeature.dev show, for any provider, what it actually supports — replacing "implements the provider contract" as an unverified claim with something a user can look up before choosing a provider.

The awkward part

Not every provider lives in an SDK-contrib repo. Many are in vendor organisations, on their own release cadence, and OpenFeature cannot run their suites — the backends and credentials are not ours. So any design has to work for providers we do not host and cannot execute.

Settled in #425

• Comparability. A report records the TCK implementation, its version and the spec revision its assets came from. The revision is a property of the TCK artifact: every implementation consumes the assets from this repository, via git submodule or, for Go, via the nested Go module, so the pin is always known. For the feature files the revision is corroborated by the Messages stream, which carries the executed source.
• Outcomes. Per-scenario outcomes come from Cucumber Messages. The envelope adds the declaration, which is an input to reading them, and a notApplicable set for capabilities a language cannot express — @numeric-coercion in JavaScript, which has no integer type. Collapsing that into "undeclared" would show every JS provider as missing something no JS provider can have.
• Known deviations. A provider can say "failed, known bug, tracked here", which is distinct from "chose not to". Three implementations had each invented a local shape for this.
• One provider, several reports. configuration keys them: flagd's RPC and in-process resolvers differ in whether they emit PROVIDER_STALE.

Never compute a conformance percentage

A single score rewards declaring fewer capabilities. Undeclare @stale and your percentage goes up. That is conformance theatre, and it exactly inverts the incentive the suite exists to create. A page should show a matrix of provider × capability with the distinct states visible, and no aggregate number. "This provider does not support configuration-change events" is genuinely useful to someone choosing a provider; hiding it behind a green 100% is actively harmful.

Open: trust

A JSON file in a vendor repository is a claim, not a fact. Three options:

Approach	Verdict
OpenFeature re-runs every suite centrally	does not scale — needs each vendor's backend and credentials
Naked self-report	worthless the moment anything is contested
Self-report with provenance	the only one that scales

Provenance meaning: the report is emitted only by CI, and carries the workflow run URL, commit SHA, and optionally a sigstore / GitHub OIDC attestation. The envelope has slots for all of these. Whatever we choose, the page should say plainly what it is showing: attested self-reports, not audits.

Open: discovery

Do not crawl. Suggested shape:

• a registry of pointers in an OpenFeature-owned repo — provider → report URL, added once by a vendor PR
• vendors self-publish the report to that URL (release asset, GitHub Pages, wherever)
• a scheduled job fetches every registered URL, validates against the schema, and rebuilds the page

The human-reviewed list stays small and stable; report content refreshes without anyone raising a PR. A /.well-known/openfeature-conformance.json convention could layer on later for vendors who prefer it.

Open: expiry

@dgenio's comment below proposes separating subject supersession (a newer provider release exists) from contract supersession (a newer TCK asset set exists), and keeping old reports visible as historical rather than deleting them. That needs the envelope to identify the tested subject strongly enough to make the comparison; provider.version is currently optional.

Suggested phasing

1. Schema in this repo. #425.
2. Emitters in each language TCK. Go, Python and JavaScript emit the earlier draft format and will be updated to the envelope once its shape is agreed. Java is in progress.
3. Registry + aggregator, designed once two or three real reports exist.
4. The page, last.

Questions

1. Should every TCK record the spec revision it vendored? Settled: the pin is always known. The envelope carries the commit and an optional release tag.
2. Is not-applicable the right fourth state? Settled as declaration.notApplicable in #425. Whether the specification should additionally scope capabilities per language, so a language-wide impossibility is recorded once rather than in every report, is still worth deciding.
3. Registry-of-pointers, or a well-known URL convention, or both?
4. How much provenance is proportionate — a CI run URL, or a full signed attestation?
5. Does a report expire? See the supersession model above.
6. Who owns the registry and the aggregator — this repo, openfeature.dev, or something new?
7. Should a report be published inside the provider's released artifact, and how strongly must the tested subject revision be identified?

Start with #425 and the provider conformance suite in Appendix F, then review the existing TCK emitters described in the issue. Focus on the unresolved questions around provenance, registry discovery, expiry and ownership. Done means reaching an agreed design for collection and trust rather than implementing the report format.