JanssenProject / JanssenProject/jans
feat(jans-cedarling): introduce format versioning for the policy store
- Dominant language
- Java
- Stars
- 647
- Forks
- 173
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 110
Description
# Introduce format versioning for the policy store
## Summary
The policy store has **no version for its own format** (the structure/schema of the
store), so any breaking change to the store schema cannot be routed or migrated
safely. This issue introduces format versioning plus a version-dispatch parsing
layer, with parse-time structures separated from runtime structures.
Motivation: the entity-type unification (see the companion issue) changes the
`token_metadata` key semantics in a breaking way and needs a migration path.
Beyond that one change, the point of the versioning layer is **maintenance**: it
gives one place where "what the file looks like" is decided, so runtime types can
evolve without every parser silently drifting out of consistency with them. A
cheaper one-off shape-sniff (e.g. "`entity_type_name` present → old shape") would
solve the immediate change but leaves the next one with the same problem.
## What "version" means today
Three different things are currently conflated, and none of them is a format version:
| Field | Meaning | Location |
|---|---|---|
| `cedar_version` | Cedar language version | `metadata.json`, `metadata.rs:50` |
| `policy_store.version` | semver of store **content** (observability only) | `metadata.json`, `metadata.rs:68` |
| legacy `version` / `policy_store_version` | same content version, Agama YAML only | `legacy_store/mod.rs:498-502` |
`policy_store.version` is surfaced at `lib.rs:605-614` (load log) and
`authz/mod.rs:1145` (decision log, `policystore_version`). It gates no behavior.
Note: `validate_legacy_metadata` (`validator.rs:221-229`, `#[cfg(feature = "tools")]`)
runs the **content** version through `MetadataValidator::validate_cedar_version`.
Confirmed misnomer/bug. Decide as part of this work whether to fix it, point it at
the real Cedar version, or drop the function.
## Proposal
### 1. Version field in `metadata.json` root
- Add the version field at the **root** of `metadata.json`, named
**`policy_store_spec_version`**. It reads as "this store conforms to vN of the
policy store specification", which is what we actually want to express — not "the
file has a shape we happen to call v2". It also does not collide with the taken
names (`policy_store.version` = content version in this format; legacy
`policy_store_version` = content version in Agama YAML, a different format that is
test-only anyway).
- Implication of that wording: **a versioned spec has to exist** to conform to. The
changelog section in the docs (see Docs) is that spec — one numbered section per
version, and the number in the file points at it. Without it the field promises
something we do not publish.
- Parse in two passes: first a minimal root struct
(`{ policy_store_spec_version: Option<...> }`)
**without** `deny_unknown_fields`, then dispatch to the version-specific parser.
No stable "envelope" abstraction is needed as long as the root stays a flat object
and the field name is stable.
- `metadata.json` is mandatory in the directory format (`loader.rs:274-280`), so the
root is a reliable place for the field.
### 2. Version constants and fail-closed policy
- `CURRENT_FORMAT_VERSION`, `MIN_SUPPORTED_FORMAT_VERSION` in one place.
- `version > CURRENT` → hard error (fail closed, no best-effort parse).
- `version < MIN_SUPPORTED` → hard error.
- Missing version → treat as the oldest supported format and emit a `WARN`
("policy store format is outdated; please update to v{CURRENT}").
**Known limitation (accepted, not solvable here):** already-released Cedarling
binaries do not know the field and `PolicyStoreMetadata` has no
`deny_unknown_fields`, so they will silently ignore it. Fail-closed only protects
readers from this version onward. In practice a breaking change should also be
structurally unparseable by the previous parser (e.g. removing a required field with
no serde default) so old binaries hard-fail instead of misreading — worth keeping in
mind when designing each future bump, but no mitigation is possible for what is
already shipped.
### 3. Parse structures separated from runtime structures
- Runtime structs must not carry `serde`. Today `PolicyStore` and `TrustedIssuer`
are already serde-free; the exceptions to split are:
- `TokenEntityMetadata` (`token_entity_metadata.rs:17`) — parsed directly by the
dir-format parser (`issuer_parser.rs`).
- `CustomIssuerMetadata` / `CustomTokenMetadata` (`custom_issuer_metadata.rs:19,32`)
— parsed directly by the parser and used at runtime.
- Per-version parse modules own `Deserialize`.
### 3.1 Version isolation
Each format version is a self-contained module owning only its own parse structs:
- Layout: `formats/v1/`, `formats/v2/`, ... Each owns its parse structs
(`Deserialize`) and parse functions.
- A version module must **not** import another version module. The only allowed
shared dependencies are version-agnostic utilities (base64, schema JSON parsing).
- The dispatch table is the **only** place that knows the full set of versions.
- Tests and golden fixtures for a version live under that version's module.
### 3.2 Migration is a chain, not N converters
Migration goes `vN -> vN+1 -> ... -> vCURRENT`, and **only the current version
converts into the runtime types**.
- Each step is a small dedicated module (e.g. `migration/v1_to_v2.rs`) depending on
exactly two adjacent version modules and nothing else.
- Consequence — the property we want: adding a field to a runtime type touches only
the current version's converter, not every version. Old versions only ever need to
know how to become the next version, which is frozen once written.
- Dropping the oldest supported version = delete its module + its migration step +
its dispatch entry, no changes elsewhere. That is the acceptance criterion for
"independent".
- Runtime types must not reference any `formats::vN` type.
### 4. Version dispatch
- The switch is **two-level**: first container/format, then `policy_store_spec_version`.
Container detection already exists (ZIP magic byte check in
`init/policy_store.rs`, `policy_store_refresh.rs`) — reuse it rather than
inventing a second one.
- One entry point: `parse_policy_store(...) -> Result`.
- All URL-ish sources (`LockServer`, `CjarUrl`, `Uri`, `ArchiveBytes`) already funnel
through the same detection, so they come along for free.
### 5. Warnings without a logger
- `load_policy_store` (`init/policy_store.rs:144`) has no logger. Follow the existing
`DefaultEntitiesWithWarns` pattern (`default_entities.rs:89-111`): the parse result
carries warnings, init reads + logs them.
- Warnings are an **enum**, not free-form strings, so they can be compared.
- The refresh path (`parse_cjar_bytes` in `init/policy_store.rs:265` and
`RefreshSource::parse` in `policy_store_refresh.rs:373`) does not go through
`load_policy_store` and has no logger, so warnings must live on
`PolicyStoreWithID` (or be returned alongside it).
- **Dedup, so periodic refresh does not spam the log:** the refresh worker keeps a
dedup set of already-logged warnings. A refresh that lands on the same format
version stays deduped (nothing re-logged); a refresh that lands on a newer format
version clears the dedup set and starts over.
### 6. Migration properties
- Migration is in-memory and per-step (see 3.2), delegating to per-component
functions (issuers / entities / policies) that are unit-testable.
- Must be lossless: unknown fields in an old store must not be silently dropped, or
the loss must be documented.
- `deny_unknown_fields` stays on parse structs, not on runtime structs.
### 7. Agama YAML is out of the versioning scheme
`FileYaml` / inline `Yaml` (`LegacyAgamaPolicyStore`) is treated as **test-only**.
- It gets no format version.
- Breaking changes are applied to it directly, and its fixtures/tests are updated in
the same PR as the change.
- Consequence: it is not a migration target and never appears in the dispatch table.
- Follow-up (separate change): decide whether to deprecate it publicly, since
`CEDARLING_POLICY_STORE_LOCAL` with `.yaml` is currently reachable by users
(`decode.rs:166-176`) even though only inline `Yaml` is documented as
"mostly testing".
### 8. Terminology
`PolicyStoreManager::convert_to_legacy` (`init/policy_store.rs:298-303`) calls the
runtime `PolicyStore` "legacy", and `legacy_store/` is the Agama format. Proposed
vocabulary to stop the collision:
- `legacy_store` / Agama YAML — keeps its name, stays outside `formats/`.
- `formats/v1`, `formats/v2`, ... — versioned directory/`.cjar` formats.
- runtime types — `policy_store`, no "legacy" in the name (rename `convert_to_legacy`).
## Note: `PolicyStoreInfo.id`
`PolicyStoreInfo.id` is documented as a content hash (`metadata.rs:60`), but nothing
computes or verifies it. It is only format-checked as hex (`validator.rs:86-87,119`)
and copied straight into `PolicyStoreWithID.id`, from where it reaches the decision
logs. In other words it is an **author-supplied opaque string that we present as a
content hash**.
Keep the field as-is for this issue — migration does not need to recompute anything,
because nothing depends on it being a real hash. But flagging it: either the docs
should stop calling it a hash, or Cedarling should actually compute/verify it. Worth
a separate issue.
## Open questions
- [ ] Version numbering: plain integer (`1`, `2`) fits "spec version N" better than
semver — confirm, plus the compatibility rules.
- [ ] What is the current number, and does introducing the field itself bump it?
If the field is introduced as v1, "missing" and "current" are semantically the
same and the WARN is misleading; if current becomes v2, v1→v2 is a no-op today.
- [ ] Should runtime `PolicyStore` keep `version: Option` (content version)
or is it parse-only?
- [ ] `validate_legacy_metadata` — fix, repoint, or delete?
## Identified code changes
- [ ] Add format-version field + validation to `metadata.json`
(`metadata.rs`, `validator.rs`, `loader.rs`).
- [ ] Add `formats/vN` modules with per-version parse structs; migration steps in
`migration/vN_to_vN+1`; only the current version converts to runtime.
- [ ] Enforce isolation: no cross-version imports; dispatch table is the only version
registry; runtime types reference no `formats::vN` type.
- [ ] Introduce `parse_policy_store` dispatch on top of the existing container detect.
- [ ] Add `CURRENT` / `MIN_SUPPORTED` constants and fail-closed errors.
- [ ] Remove `Deserialize` from runtime structs that still have it
(`TokenEntityMetadata`, `CustomIssuerMetadata`, `CustomTokenMetadata`).
- [ ] Warning enum carried on `PolicyStoreWithID`; logged from init; deduped in the
refresh worker with reset-on-newer-version.
- [ ] Apply the agreed terminology to `manager.rs` (`convert_to_legacy`).
- [ ] Decide `validate_legacy_metadata`.
## Tests
- [ ] Missing version → oldest-format parse + WARN.
- [ ] Unknown/newer version → hard error.
- [ ] Golden fixtures for each supported version.
- [ ] Version isolation: dropping the oldest version module + its migration step
leaves the others compiling and passing (import-boundary check).
- [ ] Migration chain v1 → ... → current (per step, per component, whole store).
- [ ] Refresh path surfaces warnings; same-version refresh does not re-log; newer
version resets the dedup set.
- [ ] No regressions in `archive_security_tests`.
## Docs
The format version is user-facing: a store author must be able to tell which version
their store is, what changed between versions, and how to upgrade — without reading
Cedarling source.
- [ ] **The spec + upgrade guide** in
`docs/cedarling/reference/cedarling-policy-store.md` — a "Policy store
specification versions" section. This is what `policy_store_spec_version` points
at, so it is normative, not just release notes. One numbered section per
version, each covering:
- what changed, and why (breaking vs additive);
- a before/after example of the affected files;
- the concrete steps to upgrade a store (including bumping `policy_store_spec_version`).
- [ ] Explain what the user sees in each case: missing version (WARN, parsed as
oldest), version below `MIN_SUPPORTED` / above `CURRENT` (hard error) — with the
actual log/error text so it is searchable.
- [ ] State which Cedarling release introduced / dropped support for each format
version.
- [ ] Definition of done for **every future** format bump: the PR adds its entry to
this section.
- [ ] Document `policy_store_spec_version` in the `metadata.json` section of the same page.
- [ ] `cedarling/config/default_config.yaml`.
- [ ] `docs/cedarling/reference/cedarling-properties.md`.
- [ ] Agama Lab generator emits `policy_store_spec_version` (external follow-up).
## Out of scope
- The `token_metadata` key change itself (companion issue).
- Publicly deprecating/removing Agama YAML (decision first, then a separate change).
- Making `PolicyStoreInfo.id` a real, verified content hash.
Contributor guide
Assessment
This issue has not been assessed yet.