gemaraproj / gemaraproj/gemara
Attribution and coverage gaps that break under challenge and at scale
- Dominant language
- Go
- Stars
- 61
- Forks
- 21
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 9
Description
### Is your feature request related to a problem?
---
> 🦾 Written with LLM assistance [claude-opus-5]
> 💪 Reviewed by a human before submission
---
As an operator or developer acting on a finding, I need as much of the trail as the decision warrants (what was checked, by what method, how reliably), so that I can rank it against the rest of the queue instead of treating every finding as equally urgent (gaps 2, 3, 6, and 7 in Appendix B).
As an auditor or regulator, I need to establish who produced an assessment, how, over what sample, and whether the method was ever validated, so that I can weigh the artifact on its own contents (Appendix C).
As a risk owner, I need the sampling frame and the producing executor for each assessment, so that a figure aggregated from them can carry a stated uncertainty (Appendix D).
As an incident responder, I need to select artifacts by producing instance and time window, so that a compromised scanner invalidates its own output rather than the whole corpus (Appendix A).
These range from a triage call on a normal Tuesday to an artifact challenged years later. The facts they need exist only while the assessment runs, and Gemara has no field to put them in: no producing instance, no collection method, no sampling frame, no validation status. The problem isn't that a tool left a field empty; there is no field.
### Current Project State
- `gemara-version` is required, which is exactly what a store holding years of mixed-version artifacts needs.
- `#Evidence."collected-at"` is required.
- `digest` is format-constrained rather than free text.
- `#EvidenceMapping` is a well-placed seam, and the right one to extend.
- `#EvidenceType` is an open enum, `payload` is the [CUE](https://cuelang.org/) top type, and `uri`/`url` accept any scheme. The project's instinct on extensibility looks right to me.
- The `url`/`uri` scheme widening in `b47d028` was the right call for addressing evidence where it actually lives.
### Historical Context
From the #417 discussion:
> If we think of `EvaluationLog` and `EnforcementLog` as **assertions with varying degrees of verifiability** (machine-generated assertions that say "I checked these things and here's what I found") then fields like `Provenance` increase that verifiability. **So when does an assertion become evidence?** When an auditor cites it to support a conclusion in a specific engagement expressed in `AuditLog`.
> — @jpower432, [#417](https://github.com/gemaraproj/gemara/issues/417)
Citation by an auditor is what makes an assertion *serve* as evidence. What determines whether it *survives* as evidence is the degree of verifiability the same comment names. That degree is currently unrecorded, so a consumer can't tell a high-verifiability assertion from a low one.
#417 resolved to "`#Evidence` with Provenance as a type," and what landed was the *location* half: `source: #EvidenceMapping` answers *what was consulted*. The *how* and the *by whom* were never picked up. Items 3 and 5 below are that residue.
### Cross-Examination
All citations are against `main` at `b47d028`.
| Question | What the corpus can answer today |
|---|---|
| Who ran this? | A tool name. `metadata.author` is an `#Actor`, an `#Entity` plus optional contact (`entities.cue:10-38`). Every log in a fleet carries the same one, and no field records which instance produced a given assessment. |
| When was it produced? | Not reliably. `metadata.date` is optional (`metadata.cue:42`), and an `AuditLog` whose results carry no evidence has no required timestamp at all (`auditlog.cue:60`). Under 803(6)(A) production time is an admissibility precondition, not a convenience. |
| How was it obtained? | Not recorded. `#Evidence` has no method and no originator (`auditlog.cue:82-100`). A verbatim cloud API response and a summary of a conversation are the same shape. |
| How do you know the method works? | Nothing. There is no record that an assessment method was ever validated: when, by whom, against what. |
| How much did you cover? | Not recorded. There is no sampling frame, population size, sample size, or selection method anywhere in the schema, so a scan of 40 systems and a scan of 900 are indistinguishable. |
| How reliable is that, and on what basis? | `confidence-level?` (`evaluationlog.cue:71`) offers the four `#ConfidenceLevel` values `Undetermined \| Low \| Medium \| High` (`collections.cue:51`) with no operational definition and no calibration basis. It is the producer's opinion of its own output. |
| Can I get back to the source? | Often not. `#MappingReference.url` is optional (`mapping_inline.cue:26`), and when present names a stream rather than a system, so it is frequently unresolvable from wherever the artifact ends up. |
| Does any of it still stand? | No mechanism to know. `#ArtifactType`'s thirteen values include no retraction type (`metadata.cue:71`), and `#Lifecycle` doesn't apply to logs (`collections.cue:48`). |
### Gemara Layer Impact
The premise of the layer model is that Layer 7 conclusions are rooted in Layer 5 and 6 measurements. `#EvidenceType` already permits an `AuditLog` to cite an `EvaluationLog` as evidence (`auditlog.cue:115`), which is the right design and exactly what [ADR-0022](https://github.com/gemaraproj/website/blob/main/adrs/0022-evidence-on-assessment-log.md) intended.
The consequence is that refutability climbs the same edges. If a Layer 5 measurement can't be defended, every Layer 6 and 7 conclusion resting on it inherits the defect. This is the same traversal property that makes retroactive invalidation hard, except that it fires without an incident. It's true right now, on a good day, of a single repository scanned by one Privateer plugin.
### Failure Modes
The same gaps produce an evidentiary failure under challenge and an operational failure at scale. That's why this is one issue rather than several: each argument is the other's motivation.
| Gap | Fails the question | Breaks at scale | Where |
|---|---|---|---|
| No sampling frame | "How much did you cover?" | Can't state what fraction of an estate a conclusion covers | absent |
| Unanchored `confidence-level` | "How reliable is that, and on what basis?" | Can't aggregate results across tools or networks into one figure | `evaluationlog.cue:71` |
| No evidence originator | "How was it obtained?" | Can't distinguish a cloud provider's own API response from a third party's summary of it | `auditlog.cue:82-100` |
| No per-assessment executor | "Who ran this?" | Partial compromise forces whole-document invalidation | `evaluationlog.cue:45-75`, cf. `policy.cue:124` |
| No process-validation record | "How do you know the method works?" | *(primarily evidentiary; weakest on the operational side)* | absent |
| Unresolvable references and ids | "Can I get back to the source?" | Identifiers collide once artifacts from several authorities share a store | `metadata.cue:30`, `mapping_inline.cue:26` |
| No guaranteed production time | "When was it produced?" | No reliable time-window selector; an `AuditLog` may carry no timestamp at all | `metadata.cue:42`, `auditlog.cue:60` |
| No retraction | "Does any of it still stand?" | A consumer that already holds an artifact can't learn it's void | `metadata.cue:71` |
Appendix A gives the operational column as a concrete deployment. Appendix B provides the schema forensics for each row.
Appendix A: the operational failure as a concrete deployment, with a segmented estate, a shared GRC lake, and an incident that can't be scoped
> [!NOTE]
> This deployment is invented, a composite meant to represent enterprise environments rather than a network I operate. I've kept it concrete because these gaps only surface at particular combinations of scale, topology, and elapsed time, and they're easy to talk past in the abstract. Nothing in the main body depends on it.
### Deployment Topology
Roughly 900 services across four segmented networks. A scanner fleet in each network emits `EvaluationLog`s from CI, `EnforcementLog`s at the gate, and quarterly `AuditLog`s. Artifacts replicate one-way into a central GRC data lake that also ingests from two acquired subsidiaries and a managed security service provider. I'll call those three *authorities*: separate organizations whose artifacts land in one store, each with its own identifier namespace. Retention is seven years against a records schedule (#471). The lake is the only place any of this is queryable, and it cannot reach back into the producing networks.
A shared sink on the far side of a trust boundary is a normal way to aggregate GRC data, and it's close to what Layer 7 exists to serve ([ADR-0010](https://github.com/gemaraproj/website/blob/main/adrs/0010-dual-ladder-layers.md)).
```mermaid
flowchart LR
subgraph PROD["Production estate: four segmented networks"]
SA["Segment A scanner fleet
EvaluationLog · EnforcementLog · AuditLog
runner image backdoored
2027-06-14 → 2027-11-02"]
SB["Segments B-D scanner fleets"]
end
subgraph EXT["Other feeds into the same store"]
SUB["Two acquired subsidiaries
independent id namespaces"]
MSSP["MSSP
produces for many clients"]
end
subgraph XFER["One-way replication · trust boundary"]
CA["Collector A
compromised during part of the same window"]
CB["Collectors B-D"]
end
LAKE[("Central GRC data lake
seven-year retention
no route back to producers")]
IR["IR scoping query
March 2029
Which conclusions are
still safe to rely on?"]
REG["Regulator
attestation"]
SA -->|"author = tool name
no instance, no run id"| CA
SB --> CB
CA -->|"hop not recorded"| LAKE
CB --> LAKE
SUB -->|"target.id may collide"| LAKE
MSSP --> LAKE
LAKE --> IR
IR --> REG
classDef producer fill:#eaf2fb,stroke:#4a7ab5,stroke-width:1px,color:#10243d
classDef broken fill:#fdeceb,stroke:#c0392b,stroke-width:2px,color:#5b1a12
classDef sink fill:#eaf5ec,stroke:#4f8a58,stroke-width:2px,color:#12331a
classDef consumer fill:#f2eefa,stroke:#7a5fa8,stroke-width:1px,color:#2b1d45
class SB,SUB,MSSP,CB producer
class SA,CA broken
class LAKE sink
class IR,REG consumer
style PROD fill:#fbfbfc,stroke:#c8ccd2,stroke-width:1px,color:#3d4450
style EXT fill:#fbfbfc,stroke:#c8ccd2,stroke-width:1px,color:#3d4450
style XFER fill:#fdfaf0,stroke:#d4b978,stroke-width:1px,color:#4a3600
linkStyle 0,2,4 stroke:#b8860b,stroke-width:1.5px,stroke-dasharray:5 4
```
*Red outlines mark what was compromised. Each amber dashed edge crosses a point where attribution is lost; the same information is lost on the healthy collectors and the MSSP feed.*
### Incident Scenario
A build-runner image used by part of the scanner fleet is found to have been backdoored across a five-month window, and separately the replication collector for one network was compromised for part of that same window. Either could have altered artifact content. GRC has to determine which stored conclusions are still safe to rely on, and say so to a regulator.
#### Producer Attribution
Which artifacts did the compromised producers emit? `metadata.author` is required (`metadata.cue:48`), but per the cross-examination table it names a tool class, and both `version` and `contact` are optional. There is no run, execution, invocation, or correlation identifier anywhere in the schema. The selector reduces to "every log this scanner ever produced."
#### Time-Window Selection
Which artifacts fall inside the window? Covered as gap 7 in Appendix B. For two of the three log types, a time-window filter isn't reliably available at all.
#### Intra-Log Attribution
Which assessments *within* a log came from the compromised runner? Gap 4. `#AssessmentLog` has no producer field, so partial compromise forces whole-document invalidation. At this scale that's the difference between voiding a few hundred assessments and voiding a few million.
The gaps are schema-wide, not specific to the compromised path.
Appendix B: schema forensics for each gap, with file:line citations against b47d028
Numbered to match the table in the main body.
### 1. No Sampling Frame
No population size, sample size, selection method, or stratification exists in any of the 17 `.cue` files. `#AuditLog.criteria` scopes *what* was assessed against; nothing scopes *how much of the estate*. Appendix D §1 covers why `#Result` is not a substitute, and why the default reading of that silence is the harmful one.
### 2. Unanchored `confidence-level`
`#ConfidenceLevel: "Undetermined" | "Low" | "Medium" | "High"` (`collections.cue:51`), optional on `#AssessmentLog` (`evaluationlog.cue:71`). No operational definitions, no calibration basis, not comparable across producers. `collections.cue` is `@status("stable")`. Full treatment in Appendix D §3.
### 3. Missing Evidence Originator and Method
`#Evidence` is `{id, type, collected-at, payload?, source?, description?}` (`auditlog.cue:82-100`). `collected-at` is required, which is good and load-bearing. But nothing records *who produced the payload*, as distinct from who collected it, and nothing records *how it was obtained*.
A verbatim cloud-provider API response and a summary of a conversation are the same shape, and a consumer cannot tell them apart. That is the entire reliability hierarchy (external over internal, observation over inquiry) collapsed into one undifferentiated field. `#EvidenceType` (`auditlog.cue:115`) is an open enum and could carry a hint by convention, but it types the *kind of thing*, not its provenance or the procedure that yielded it.
### 4. Missing `#AssessmentLog` Producer
`#AssessmentLog` (`evaluationlog.cue:45-75`) has no producer field, and `metadata.author` is document-level and singular. At fleet scale a single `EvaluationLog` aggregates work from many executors, which was the stated goal of #170 and #23.
`policy.cue:124` gives `#AcceptedMethod` an `executor?: #Actor`, so a *plan* can declare who is expected to run an assessment. Nothing on the measurement side records who actually did.
`steps` is the closest thing to a producer record, and it still isn't one. `#AssessmentStep` is `string` (`evaluationlog.cue:77`); `#EnforcementStep` is `string` (`enforcementlog.cue:50`), documented as "the code paths or addresses that carried out this enforcement action" (`enforcementlog.cue:42`). In the repo's own test data these hold values like `github.com/revanite-io/pvtr-github-repo/evaluation_plans/...`. The intent to record what ran is already there. It's just unstructured, so it can't be resolved, grouped, or queried.
### 5. No Process-Validation Record
Grepping for validation, calibration, accuracy, verified, or certified across all 17 `.cue` files returns nothing relevant. There is no field asserting that an assessment method was checked against a known-good reference, when, by whom, or with what result. This is the field an authenticity challenge asks for first (Appendix C).
### 6. Unresolvable References and Identifiers
#### Reference Resolution
`#EvidenceMapping.reference-id` resolves to a `#MappingReference`, whose `url` is optional (`mapping_inline.cue:9-27`). Even when present, a URL names a *stream*, not a system: no endpoint identity, tenancy, or credential. `https://scanner.corp.internal/...` is unroutable from a lake and may collide with a different system in the lake's own namespace. `digest` is optional (`mapping_inline.cue:76`) and pins *content*, not *channel*.
That leaves two options. Inline the payload and you carry size and sensitive data across a trust boundary, the thing [ADR-0022](https://github.com/gemaraproj/website/blob/main/adrs/0022-evidence-on-assessment-log.md) warns against. Reference it and the far end can't resolve it. Neither leaves a record that the reference *was* resolved and verified at time T by party P while it was still reachable.
#### Identifier Namespacing
There is no tenant, trust-domain, originating-authority, or namespace field anywhere; the only hits for those words are prose comments. `metadata.id` is a bare `string` with no format and no uniqueness constraint (`metadata.cue:30`), while `uri`, `url`, `digest`, and `email` all carry regexes. `#Entity.id`'s comment says it "uniquely identifies the entity" (`entities.cue:11-12`) without stating the scope of that uniqueness.
All 14 `_unique*` constraints in the repo are document-scoped: `_uniqueResultIds` (`auditlog.cue:29`), `_uniqueRefIds` (`metadata.cue:63`), `_uniqueControlIds` (`controlcatalog.cue:20`), and eleven more, each iterating a list inside one artifact. None expresses uniqueness *across* artifacts. That is correct and sufficient for a single producer. In a store holding several authorities it means there is no primary key, and the operator must synthesize a namespace out of band from the ingestion path. That is exactly the metadata that gets lost across seven years and a re-platforming.
The cross-layer traversal edges already exist: `#Evidence.source` → `#EvidenceMapping.reference-id` → `#MappingReference.id` → another artifact's `metadata.id`. The graph is expressible today. It just doesn't resolve across authorities. Fixing identifiers fixes traversal; no new traversal machinery is needed.
### 7. No Guaranteed Production Time
`#Log` is `{metadata, target}` and nothing else (`collections.cue:39-45`). `metadata.date` is optional (`metadata.cue:42`). Only eight distinct fields in the whole schema carry a `#Datetime`, and the guaranteed ones sit inside optional structures.
- `#EnforcementLog` always has at least one, since `ActionResult.start` is required (`enforcementlog.cue:37`).
- `#EvaluationLog` has one unless every result was `Not Run`/`Unknown`/`Not Applicable` (`evaluationlog.cue:35-42`).
- `#AuditLog` whose results carry no evidence has no required timestamp at all. `evidence` is optional on `#AuditResult` (`auditlog.cue:60`), and `Evidence."collected-at"` (`auditlog.cue:90`) is the only other datetime it could have had.
So only `EnforcementLog` is unconditionally selectable by time window; for the other two it depends on the contents. Even a time-window filter, the cheapest possible selector, isn't reliably available at the top layer. Per Appendix C, production time is also an admissibility precondition under 803(6)(A), which makes this more than a query convenience.
### 8. No Retraction
`#ArtifactType`'s thirteen values include no retraction type (`metadata.cue:71`). `#Lifecycle` (`Active | Draft | Deprecated | Retired`) applies to guidelines, controls, and assessment requirements, not to logs (`collections.cue:48`).
Supersession exists, but only on the definitional side: `replaced-by` links a deprecated or retired guideline, control, or assessment requirement to its successor (`guidancecatalog.cue:105`, `controlcatalog.cue:65`, `controlcatalog.cue:86`). Those are the same three kinds `#Lifecycle` covers, and logs have no equivalent. And supersession isn't invalidation: "this was replaced" and "this was never trustworthy" are different claims with different consequences downstream. There is no `invalidated-by` link anywhere.
Combined with an unconstrained `metadata.id`, a producing org can't unambiguously *name* the artifact it wants to void, and a consumer that already ingested it has no in-band way to learn it's void.
### Deliberate Exclusions
#### Chain of Custody
Nothing records that an artifact crossed a diode or an ETL path, was re-batched, re-signed, or redacted. I think this belongs to the sink's own ingestion metadata rather than the artifact. Anything recording what happened *after* the producer released it must be appended afterwards, and a document that gets appended to can't be signed once. OCSF's `loggers[]` is the right model, for the reason given in Appendix F.
#### Entity Relationships
`#Log.target` is exactly one `#Resource` (`collections.cue:44`), and `#Resource` has no part-of, runs-on, or depends-on (`entities.cue:41-49`); `environment` is free text. N logs give N disconnected nodes. I think Gemara should reference an external inventory by stable identifier rather than grow a graph model (open question 6).
Appendix C: what a regulator or a court actually requires, and which requirement each gap fails
I'm not a lawyer. I'm citing the US Federal Rules of Evidence because they're the most explicit written articulation of what it takes for a record to carry weight; EU and UK regimes differ in mechanism but impose the same structural demands. These requirements have been written down for decades, they're stable, and they name exactly the fields Gemara doesn't have.
A Gemara artifact offered to support a compliance position has to clear three independent hurdles. Most discussion of "provenance" addresses only the first, and clearing it doesn't help with the others. The advisory committee is explicit that "Rule 902(13) is solely limited to authentication, and any attempt to satisfy a hearsay exception must be made independently."
| Hurdle | The requirement, verbatim | What it demands of the record | Gemara today |
|---|---|---|---|
| Authentication: is this what it claims to be? | [901(a)](https://www.law.cornell.edu/rules/fre/rule_901): "the proponent must produce evidence sufficient to support a finding that the item is what the proponent claims it is" | An identity claim someone can test | `metadata.author` names a tool class, not an instance |
| Authentication of machine output | 901(b)(9): "Evidence describing a process or system and showing that it produces an accurate result" | A description of the process *and* evidence it is accurate | No process-validation record exists. This is gap 5. |
| Self-authentication | [902(13)](https://www.law.cornell.edu/rules/fre/rule_902): "A record generated by an electronic process or system that produces an accurate result, as shown by a certification of a qualified person" | A certification, by a named qualified person, about the process | No analogue; see below |
| Copied data | 902(14): "if authenticated by a process of digital identification, as shown by a certification of a qualified person" | A certification that someone *checked* it, beyond the hash itself | `digest` exists; nothing records that anyone verified it, or who |
| Hearsay exception | [803(6)(A)](https://www.law.cornell.edu/rules/fre/rule_803): "made at or near the time by — or from information transmitted by — someone with knowledge" | Production time is a precondition, not a convenience | `metadata.date` is optional (`metadata.cue:42`); an `AuditLog` may carry no timestamp at all |
| | 803(6)(C): "making the record was a regular practice of that activity" | Scheduled recurring collection, distinguished from one-off | Inexpressible |
| | 803(6)(E): "the opponent does not show that the source of information or the *method or circumstances of preparation* indicate a lack of trustworthiness", so the record comes in unless they do | Method and circumstances are the named attack surface | Neither is recorded |
| Opinion reliability | [702(b)](https://www.law.cornell.edu/rules/fre/rule_702): "based on sufficient facts or data" | Coverage and sampling | Gap 1 |
| | 702(c): "the product of reliable principles and methods" | A method with known properties | `confidence-level` is not that; gap 2 |
### Rule 902(13): Certification Artifact
It is also a better-shaped extension point than an open slot. The rule contemplates a certification by a qualified person that a process produces accurate results, and the committee note says it "allows the authenticity foundation that satisfies Rule 901(b)(9) to be established by a certification rather than the testimony of a live witness." That is a specific, bounded, machine-representable document with a defined purpose and a defined signer. It is far more useful to standardize than a generic slot for arbitrary identity systems, because a consumer knows what to do with it.
### Digest Sufficiency
The 902(14) committee note is worth reading in full: "identical hash values for the original and copy reliably attest to the fact that they are exact duplicates. This amendment allows self-authentication by a certification of a qualified person that she checked the hash value of the proffered item and that it was identical to the original." The evidentiary act is the *checking*, by an identified person, at a time. Gemara records the input to that act and not the act. This is the same shape as gap 6: an unresolvable reference means the check can never be performed.
### Rule 803(6)(E): Method as Attack Surface
The burden sits with the opponent, which sounds favourable, but note what they're invited to attack: "the method or circumstances of preparation." An opponent doesn't have to prove the scan was wrong. Per the committee note, they are "not necessarily required to introduce affirmative evidence of untrustworthiness." It is enough to argue from circumstances. If the corpus cannot state the method, the circumstances, the executor, or the coverage, there is nothing to rebut with.
This also lines up with ordinary audit doctrine, which ranks evidence by reliability along axes Gemara can't currently express: externally sourced above internally generated, the auditor's direct observation above the entity's representation, contemporaneous above reconstructed. That hierarchy is exactly what OSCAL's required `observation.methods` (`EXAMINE | INTERVIEW | TEST`) encodes, which is a good reason to borrow it rather than invent one (Appendix F).
### Scope Limits
Not that Gemara make artifacts court-admissible. Admissibility turns on facts no schema can supply, and on a judge. Only that the schema stop *foreclosing* it by having nowhere to put the foundational facts. Right now a producer who wants to do this properly has no field to do it in.
Appendix D: why the corpus can't currently support a risk figure, covering sampling, correlation, and aggregation
The layer model exists so that Layer 6 and 7 conclusions can be built from Layer 5 measurements. That is an aggregation, and every aggregation needs two things the corpus doesn't carry: *what fraction of the population was covered*, and *how correlated the observations are*. Without either, you can state a number but you cannot state its uncertainty. A number whose uncertainty can't be stated can't be tested, which makes it unfalsifiable rather than merely imprecise.
### 1. Sampling Frame
There is no population size, sample size, selection method, or stratification anywhere in the schema. `#Result` (`evaluationlog.cue:79`) includes `Not Run`, `Unknown`, and `Not Applicable`, but those are per-item outcomes, not a frame. They tell you an individual check didn't run, not that 860 systems were never in scope to begin with.
Selection method is what licenses an inference, and the methods aren't interchangeable:
| Selection | What it licenses |
|---|---|
| Complete enumeration | A statement about the population, directly |
| Random / stratified random | A statement about the population, with an interval |
| Risk-weighted | A statement about the high-risk stratum; says little about the rest |
| Judgmental / convenience | A targeted probe. Licenses *no* statistical inference about the population |
Absent a frame, the only reasonable reading of a Gemara log is that it covers what it appears to cover: a census. That reading is usually wrong and always overstates coverage. So the gap doesn't produce caution in consumers; it produces unwarranted confidence, silently.
### 2. Correlation Structure
Assessments produced by one executor are not independent observations. They share a tool version, a configuration, a credential set, a set of bugs. If that executor is wrong, it is wrong across all of them at once.
The standard treatment is Kish's [design effect](https://en.wikipedia.org/wiki/Design_effect). For `m` observations per cluster with intra-cluster correlation `ρ`:
```
DEFF = 1 + (m − 1)ρ
n_eff = n / DEFF
```
For assessments from a single executor, `ρ` is close to 1, which is what "same tool, same bug" means. So a log carrying 400 assessments from one runner has an effective sample size near 1, not 400. Any interval computed as though `n = 400` is too narrow by roughly a factor of 20.
`#AssessmentLog` has no producer field (`evaluationlog.cue:45-75`), so a consumer cannot cluster the observations, cannot estimate `ρ`, and therefore cannot compute `n_eff`. **Every aggregate derived from a multi-executor Gemara corpus today is overconfident by an unknown factor.**
This is not an incident-response argument: it holds for a perfectly healthy fleet, no compromise required. Incident scoping is a second, lesser benefit of the same field. This is why I've ranked the executor field above retraction and well above signatures.
### 3. `confidence-level` as Unanchored Ordinal
Five problems with the field as defined (Appendix B §2), any one of which would be disqualifying for aggregation:
1. No operational definition. Nothing states what distinguishes `High` from `Medium`, so it means whatever each tool author felt.
2. Not comparable across producers. Tool A's `Medium` and tool B's `Medium` are unrelated quantities. In a multi-source store they will be aggregated anyway.
3. Ordinal, so arithmetic is undefined. There is no meaningful average of `High` and `Low`, but the shape of the field invites one.
4. Self-asserted, never calibrated. Nothing tracks whether assessments marked `High` are in fact right more often. A confidence claim that is never scored against outcomes is not a confidence claim.
5. It conflates two different uncertainties: did the check execute correctly (measurement), and does passing the check imply the control is effective (construct validity). These have different sources and different remedies, and collapsing them means neither can be addressed.
Any of these would be a large improvement, and I don't have a strong preference:
- Anchor it: publish operational definitions per level, and require a calibration basis.
- Replace it: record a stated error characteristic of the method (false-positive and false-negative rates, or a validation reference), which is what 702(c) and Appendix C are really asking for.
- Fence it: document explicitly that it is a subjective producer annotation that MUST NOT be aggregated or compared across producers. This costs nothing and immediately stops the field from doing harm.
The field is `@status("stable")`, so option three may be the only near-term one.
### 4. Validity Periods and Decay
Not a criticism of #451. It is an open draft, and it solves the plan-side problem it set out to solve. But a binary valid-until-T encodes a specific and false model: hazard is zero until T, then infinite. Configuration drift behaves much more like a continuous process, and the right decay rate differs per control and per environment, so it could never be a schema constant anyway. Declaring it in the plan rather than recording it at the measurement makes that harder, not easier, since the plan can't know what the measurement will encounter.
I'm not asking Gemara to model decay. I'm asking it to record the inputs (measurement time, method, and what change-detection the measurement was based on) so that someone downstream can fit a decay curve instead of inheriting a step function they didn't choose.
### Scope Limits
Not a statistics engine. Gemara should not compute intervals, fit curves, or aggregate anything. Not a mandatory sampling frame on every artifact either: a single-repository scan is a census and can say so in one field, cheaply. Only that the facts an aggregation needs be *recordable*, so that the consumers who do aggregate can do it correctly, and so that the ones who don't can tell they shouldn't.
### Describe the solution you'd like
Gemara's job is to make evidence *assessable*, not *conclusive*. Record the facts a risk assessor or an auditor needs in order to weigh an artifact: origin, method, executor, sampling frame, validation status, and time. Don't assert that the evidence is good. By that test `confidence-level` is currently worse than no field, because it asserts goodness without an anchor and invites reliance on it.
Priority order below is by refutability: what a skeptical reader attacks first.
- [ ] 1. Sampling frame. An unstated sample makes a conclusion uninterpretable. No prior art in the project at all.
- [ ] 2. Anchor or retire `confidence-level`. Currently `@status("stable")`, so this is harder than anything else here and worth starting early.
- [ ] 3. Evidence origination and method. Who produced the payload, distinct from who collected it; and how it was obtained. OSCAL and CycloneDX both model it (Appendix F), so this is a compatibility gap rather than a new invention.
- [ ] 4. Per-assessment executor. `#AssessmentLog` has no producer field while `#AcceptedMethod` has `executor?: #Actor` (`policy.cue:124`). Optional and additive. Prerequisite for #170.
- [ ] 5. Process-validation record. The foundation an authenticity challenge asks for, and the honest shape for an extension point. CycloneDX's `declarations.evidence[].reviewer` is the nearest existing analogue.
- [ ] 6. Resolvable identifiers; guaranteed production time; scheduled-vs-ad-hoc. The first is the only genuinely breaking change in this list.
- [ ] 7. Retraction. Its own artifact type, not a mutable field. Depends on 6: you can't void what you can't name.
Deliberately last: signatures and attestation envelopes. They answer *who asserted this*, which is not the question above, and an adversary won't reach them while items 1 and 2 are open. Appendix E covers where that work should live instead.
Items 1 and 2 are larger than schema fields. They're a statement about evidentiary sufficiency, and they may be more than the project wants to carry. I'd rather put the full requirement forward and let maintainers scope it than pre-scope it myself and lose the argument for why the rest matters.
I'd like the direction recorded as an ADR, with narrow implementation issues underneath it.
Appendix E: compatibility cost, the oasdiff gate, the temporary .oasdiff-allow, and what surprised me
### Breaking-Change Cost
The gate runs producer-safe as well as consumer-safe (`CONTRIBUTING.md`, "Schema compatibility"): a request "may not add a new requirement that existing writers cannot satisfy." So a required core needs either a deliberate allowlisted exception or a `@v2` lineage. Optional additive fields (items 3, 4, and 5 in the ask) should be free, but I'd run `make breaking-check` rather than assume, including for adding an enum value to `#ArtifactType` for item 7.
### The `.oasdiff-allow` Exception
It scopes itself to a one-time projection-fidelity transition and says so in its own header (`.oasdiff-allow:1-12`):
> These exceptions cover ONLY that transition. Once this PR merges and the next v1 release publishes a faithful openapi.yaml, the baseline matches the projection and a follow-up PR MUST delete this file.
Reusing it for a required core would quietly convert a temporary file into a permanent one. So this is really a question about establishing a *standing* exception policy, or accepting a major-version bump (open question 3).
### Experimental Status and Compatibility
I assumed the experimental schemas were cheaper to change. They aren't. `extractFileStatus` (`cmd/internal/cmd/converter.go:195-224`) reads the annotation and the parser stamps it onto every emitted schema as `x-status` (`converter.go:61`, and at `:298`, `:316`, `:339`, `:351`, `:361`, `:368`). It's projected as documentation and nothing more. `breaking-check` (`Makefile:77-95`) diffs the whole projection against the latest `v1` baseline with no status filtering, and CI runs it on every PR to `main` (`.github/workflows/ci.yml:28-29`).
In the current `v1.5.0` baseline asset, 59 of 92 projected schemas carry `x-status: experimental`. Roughly two thirds of the published contract is nominally experimental and fully frozen.
#451 is a live example. It is a draft PR that changes `#AssessmentPlan.frequency` from string to int and removes `evidence-requirements`, and describes itself as ":warning: Breaking changes to experimental schema ONLY." Both of those fields are present in the `v1.5.0` baseline, under `AssessmentPlan` with `x-status: experimental`. Its CI is green, but the run contains no "Check breaking changes" step at all, because the branch head is from 2026-08-07 and the step landed on 2026-08-27 in #460. It has never run the gate, and will fail it on rebase.
So the belief that experimental schemas are cheaper to break is held in the project but isn't backed by the tooling. I think that's worth raising on its own regardless of what happens to this issue: either `@status` should gate the check, or it should stop implying a stability distinction it doesn't deliver.
It would not buy this issue much either way, because most of these gaps have to be fixed in a stable file. Only gap 3 lands squarely in an experimental one (`#Evidence`, in `auditlog.cue`). Gaps 2, 4, 6, 7, and 8 land in `collections.cue`, `evaluationlog.cue`, `metadata.cue`, and `mapping_inline.cue`, all four `@status("stable")`, and `ConfidenceLevel`, `AssessmentLog`, `Metadata`, `MappingReference`, and `ArtifactType` all project as `x-status: stable`. Some of them cite experimental schemas in passing, `#AcceptedMethod` in `policy.cue` for gap 4, but that's the precedent, not the field being added. So the concession the annotation appears to offer isn't available for most of what's asked here, which makes open question 3 the one that actually governs the cost.
### Strict Variants in the OpenAPI Projection
`#_EvidenceStrict` and `#_AssessmentLogStrict` are enforced in CUE only. The projection rewrites hidden `#_` definitions to their visible base and drops the conditional requirements, by design (`cmd/internal/cmd/converter.go:43-48`):
> Rewriting a ref intentionally drops the hidden definition's extra constraints (e.g. conditional requiredness), which OpenAPI cannot express; the ref collapses to the visible base type.
So consumers generating types from the published `openapi.yaml` don't see the payload-or-source rule at all. That's the rule raised in #470 and enforced by PR #475, which is `b47d028`, the commit this issue is written against. Any conditional requirement proposed here inherits the same problem, which is open question 4. It's also an argument for expressing the core as plain required fields where possible rather than as conditional constraints, even though conditionals would be the more precise modelling.
### Describe alternatives you've considered
The supply-chain attestation ecosystem solves *authorship* well and *reliability* only in part. OSCAL, which Gemara already claims compatibility with, models most of the evidentiary core; CycloneDX 1.6 models more of it than its reputation suggests. Sampling and coverage are absent from every format checked.
### OSCAL Compatibility
OSCAL models most of the evidentiary core and Gemara has nowhere to put any of it: `observation.methods` (required, minimum one), `observation.origins[].actors[]`, `observation.subjects[]`, and `relevant-evidence[].href`. ADR-0002 already commits the schemas to OSCAL compatibility, so items 3 and 5 in the ask are closing a commitment rather than inventing something. The tooling corroborates it: `gemara2oscal` (#74) converts the definitional layers and stops exactly where evidence begins, which is open question 5. Appendix F has the field-by-field comparison.
### Custom Payloads
`#Evidence.payload` is the CUE top type and `#EvidenceType` is an open enum (`auditlog.cue:115`), so a producer can already embed an SBOM, an attestation, or a bespoke record carrying executor, method, and coverage. On that reading the facts are available today and no schema change is needed.
The facts would be present without being addressable. Every operative question in this issue is a selection over a corpus: which artifacts a compromised instance emitted, which fall inside a window, which assessments cluster under one executor. A predicate over an arbitrary payload needs a parser per producer, and a lake ingesting from several authorities needs one for each, maintained across the retention period. Nothing in the artifact declares which shape a payload carries, so a consumer cannot dispatch to the right parser without out-of-band knowledge, which is exactly the knowledge that does not survive seven years and a re-platforming.
The objection is strongest with CycloneDX, and it lands in part. CycloneDX 1.6 `declarations` carries an assessor with a `thirdParty` flag, per-claim `reasoning` and `counterEvidence`, and evidence with `author`, `reviewer`, `created`, and `expires`; `componentIdentityEvidence` carries a `methods[].technique` enum and a numeric `confidence`. A payload of that shape really would supply an originator, a reviewer, a method, and an anchored confidence. What no SBOM format supplies at any version is a sampling frame. Appendix F has the full comparison and the limits.
SPDX carries less than its reputation suggests here. `creationInfo.creators` and `creationInfo.created` are both required, but `creators` identifies "who (or what, in the case of a tool)" by name and version, which is a tool class rather than the instance that ran. That is the same gap as `metadata.author`.
The SBOM ecosystem is the precedent for this ask rather than an argument against it: formats that were largely optional needed a mandated minimum bolted on afterwards (Appendix F, "Status Quo").
Two mechanical constraints close the option independently of any of that. ADR-0022 warns against inlining payloads across a trust boundary (Appendix B §6), and #468 records that `payload` does not survive the OpenAPI projection intact. The escape hatch is neither portable nor guaranteed to arrive.
Appendix F: prior art in detail, covering OSCAL, CycloneDX, OCSF, in-toto/SLSA/DSSE, Sigstore, SCITT, SPIFFE, GUAC, and OpenVEX
I've built this table on reliability rows rather than the supply-chain rows a provenance discussion usually uses, because those rows concede the framing before the comparison starts.
| | OSCAL | CycloneDX 1.6 | OCSF v1.9 | in-toto / SLSA | Sigstore | SCITT | Gemara today |
|---|---|---|---|---|---|---|---|
| Evidence originator, distinct from collector | `origin.actors[]` `{type, actor-uuid}` | `declarations.evidence[].author`, and `reviewer` beside it | `metadata.product` | `runDetails.builder.id` | Fulcio cert claims | `iss` (mandatory) | `metadata.author` (tool name) |
| How evidence was obtained | `observation.methods` (required) | `componentIdentityEvidence.methods[].technique` enum; free text in `declarations` | none | none | none | none | absent |
| Sampling / coverage | none | none | none | none | none | none | absent |
| Method error rate | none | `componentIdentityEvidence.confidence` 0-1, self-asserted | none | none | none | none | `confidence-level` (unanchored) |
| Process validation | none | `declarations.evidence[].reviewer`; `assessors[].thirdParty` | none | none | none | none | absent |
| Subject binding | `observation.subjects[]` | `declarations.targets[]` | `device`, `resources[]` | digest only | digest only | `sub` | `target` (unscoped id) |
| Resolvable evidence reference | `relevant-evidence[].href` + back-matter hash | `externalReferences` | none | `ResourceDescriptor` | none | detached payload | `url?` (optional) |
| Namespace across authorities | UUIDs throughout | `serialNumber` per document | `tenant_uid` | none | cert issuer | `iss` | absent |
| Retraction | `revisions[]` (lineage, not retraction) | `evidence[].expires`, `claims[].counterEvidence` | none | none | *none by design* | append a later statement | absent |
| Authenticity | none | JSF `signature` at several levels | Record Integrity profile | DSSE | keyless + Rekor | COSE_Sign1 | absent |
Schema-checked against the [SPDX 2.3](https://spdx.github.io/spdx-spec/v2.3/) and CycloneDX 1.6 and 1.7 JSON schemas rather than their prose documentation.
One row is empty everywhere. Sampling and coverage appear in no format at any version: neither the SPDX 2.3 nor the CycloneDX 1.6 schema defines a field for a sample, a coverage figure, a population, or a stratification, and OSCAL, OCSF, in-toto, Sigstore, and SCITT carry no analogue. Item 1 in the ask has no prior art anywhere, which is a weaker position to argue from than the others and the reason it is ranked first rather than assumed.
### [OSCAL](https://pages.nist.gov/OSCAL/learn/concepts/layer/assessment/assessment-results/)
`observation.methods` is required, minimum one, with allowed values `EXAMINE | INTERVIEW | TEST | UNKNOWN` and locally-definable extensions: "identifies how the observation was made." Alongside it: `observation.origins[].actors[]` ("the actor that produces an observation... one or more actor type can be used to specify a person that is using a tool"), `observation.subjects[]` ("identifies who was interviewed, or what was tested or inspected"), and `relevant-evidence[].href` as a resolvable reference with hashes via back-matter `rlink/hash`.
ADR-0002 says the schemas "should be 100% OSCAL-compatible while optimizing for automation." Each row above where OSCAL has a field and Gemara has nowhere to put it is a compatibility gap against that commitment.
And a checkable consequence, which I did check. [`complytime-labs/gemara2oscal`](https://github.com/complytime-labs/gemara2oscal) (#74) converts *only the definitional layers*: Layer 1 `GuidanceDocument` → OSCAL `Catalog` and `Profile` (`controls/controls.go`), and Layer 2 `Catalog` → `ComponentDefinition` (`component/builder.go`). It emits no `AssessmentResults`, constructs no `Observation`, and does not read `EvaluationLog`, `EnforcementLog`, or `AuditLog` at all.
I don't know whether the measurement side is unmapped because it can't currently be populated or simply because nobody has done it yet, and that's open question 5. Either way, OSCAL's required `observation.methods` is one concrete reason the second half would be hard to write today.
Against OSCAL generally: it's heavyweight, #74 already records a "stuck in time" concern, and `revisions[]` gives version lineage rather than invalidation.
### [CycloneDX](https://cyclonedx.org/docs/1.6/json/)
The closest prior art in the supply-chain family, and stronger than a provenance discussion usually
credits it for. Two blocks matter.
`declarations` (added in 1.6) is a compliance attestation model: `assessors[]` with a `thirdParty`
boolean, `attestations[]` mapping requirements to claims, `claims[]` carrying `predicate`,
`reasoning`, `evidence`, and `counterEvidence`, and `evidence[]` carrying `author`, `reviewer`,
`created`, `expires`, and a signature. The `author`/`reviewer` split is the originator-distinct-from-
collector distinction item 3 asks for, and `assessors[].thirdParty` is a coarse encoding of the audit
axis in Appendix C that ranks external above internal.
`componentIdentityEvidence` carries a `methods[].technique` enum (`source-code-analysis`,
`binary-analysis`, `manifest-analysis`, `ast-fingerprint`, `hash-comparison`, `instrumentation`,
`dynamic-analysis`, `filename`, `attestation`, `other`) with a numeric `confidence` from 0 to 1 per
method and overall. That is a better-shaped version of `confidence-level`: still self-asserted and
uncalibrated, but anchored to a scale and attached to a named technique rather than floating free.
Three limits keep it from answering this issue. The scope is component identity within a build
artifact and conformance claims about a target, not a measurement of a running system, so
`methods[].technique` enumerates ways to identify a dependency rather than ways to assess a control.
`declarations.evidence[].description` records "how it was created" as free text, so the method is
only enumerable in the component-identity case. And there is no sampling frame at any level.
Worth borrowing regardless: the `author`/`reviewer` pair, `thirdParty`, and confidence-attached-to-
technique are all cheap to mirror and already have an installed base.
### [OCSF](https://schema.ocsf.io/)
The best fit for the lake half of the problem, since it was designed for many producers feeding one queryable store. `loggers[]` is an ordered array describing each hop in the event pipeline (a chain-of-custody model with no Gemara analogue), and `logged_time`/`processed_time` separate ingest from occurrence. `original_event_uid` exists specifically for traceability back to the raw entry.
But it works precisely *because* OCSF is a normalization target at ingest rather than a producer contract. That's why I think the right conclusion is a Gemara→OCSF mapping at the sink instead of OCSF fields in the CUE. OCSF is event-shaped and Gemara is document-shaped; if this belongs anywhere in Gemara it's at the Layer 6/7 aggregation boundary.
### in-toto, SLSA, DSSE, and Sigstore
Worth doing, and worth being precise about what it buys.
It solves document authenticity (DSSE), producer identity for CI-originated artifacts (Fulcio certificate claims), and, best in class, establishing origin years later after keys have rotated, via short-lived certificates bound to a transparency-log entry. That last one is a genuinely hard problem and Sigstore's answer is better than anything Gemara could build.
Three limits:
1. The subject model doesn't fit. in-toto binds a `subject` by digest, which works for an immutable build artifact. A running system has no digest. Wrapping a Gemara log makes the log's own bytes the subject, attesting to authorship of the document rather than to the system it describes.
2. No retraction, by design. [Rekor](https://docs.sigstore.dev/logging/overview/) is append-only: "the log remains append-only and entries are never mutated or removed." There is no revocation and no supersession semantics.
3. Bundles are not a custody vehicle. The [Bundle spec](https://github.com/in-toto/attestation/blob/main/spec/v1/bundle.md) says so outright: "The Bundle is not authenticated as a whole... an attacker might be able to delete valid attestations, replay obsolete attestations, and/or inject invalid or irrelevant attestations in a Bundle without being detected." So retraction discovery has to be pull-from-log, not push-in-bundle.
One thing worth borrowing regardless: `ResourceDescriptor` (`name`, `uri`, `digest`, `content`, `downloadLocation`, `mediaType`, `annotations`) is a strictly richer `#EvidenceMapping`, and aligning field names would be cheap.
### SCITT
Mandatory `CWT_Claims {iss, sub}` in the protected header is exactly the producer-identity-plus-namespace core, already mandatory, already at the right layer. The ledger is append-only and non-equivocating with receipts as inclusion proofs, the payload is content-agnostic, and detached payloads solve the inline-versus-reference dilemma directly. Retraction is register-a-later-statement against the same `sub`, which is the correct semantics: you never delete a claim, you append a contradicting one.
Against: it is COSE/CBOR-centric, implementations are thin, and Gemara's current audience is YAML-and-Go. The maturity objection I'd have made a year ago has gone: the architecture was published as [RFC 9943](https://www.rfc-editor.org/rfc/rfc9943.html) in June 2026, a Proposed Standard on the IETF stream, so draft-22 was the last Internet-Draft rather than the current state. The API layer (`draft-ietf-scitt-scrapi`) is still a draft. I'm noting it as the direction the ecosystem is heading rather than proposing it.
### SPIFFE
The trust domain is a real namespace and `spiffe://` is a real workload identity, so it's a good thing to be *able* to record in a producer-identity field. But the [SPIFFE-ID spec](https://github.com/spiffe/spiffe/blob/main/standards/SPIFFE-ID.md) is explicit about the limit (§4.1.1, Temporal Accuracy): "While it is generally the case that assertions in SVIDs are true at the time of issuance, it does not necessarily mean that they are true at the time of use." SVIDs are deliberately short-lived, bundles rotate, and SPIRE models current state rather than history. A `spiffe://` string recorded in 2027 may be unresolvable or reassigned by 2034, which is the window this issue is about. It also wouldn't cover GitHub Actions OIDC or human auditors. The case is for an opaque, scheme-agnostic identity field, not for SPIRE as the mechanism.
### GUAC
[GUAC](https://guac.sh/) (OpenSSF incubating) already ingests attestations into a queryable graph and answers blast-radius questions over GraphQL. Given Appendix B §6, Gemara needs no new traversal machinery, just identifiers that resolve. After that it's someone else's solved problem.
### [OpenVEX](https://github.com/openvex/spec/blob/main/OPENVEX-SPEC.md)
Comparable in scale and ambition to Gemara, and it made exactly the choice I'm proposing. Its required core is tiny: `@context`, `@id` (an IRI, not a bare string), `author` ("should ideally be a machine readable identifier such as an IRI"), `timestamp`, and `version`, on a document that is otherwise largely optional. And it explicitly declines to grow a signature field, deferring instead: `author` identity "SHOULD be cryptographically associated with the signature of the VEX document or other exchange mechanism."
It also handles supersession by time ordering rather than a revoke verb, as "a sequence of statements, each overriding, but also enriching the previous ones with new information." That is a lighter-weight model for item 7 than a formal retraction type, and worth considering as an alternative.
### Status Quo: Optional Fields Only
Lowest cost, keeps the compatibility gate happy, lets each deployment do what it needs. The concern is that fields consumers can't rely on tend not to get populated, sinks diverge in how they backfill, and artifacts become less portable, which is some of what the shared schema is buying. The SBOM ecosystem went through this: formats that were largely optional needed a mandated minimum bolted on afterwards (NTIA's seven minimum elements) to become useful to consumers, and doing that later was more disruptive than doing it early. That's the main reason I'd rather raise it now.
Optional is worse than absent for `confidence-level` (Appendix D §3): a field that is present, unanchored, and inconsistently populated actively misleads.
### Success Criteria
The cross-examination questions become answerable from the corpus alone, without out-of-band knowledge:
1. Who ran this: identify the producing instance rather than the tool class, for any given assessment within a log.
2. When it was produced: obtain a guaranteed production timestamp on every log type, including those whose results carry no evidence.
3. How it was obtained: distinguish direct machine observation from a third party's report from a human's statement, and identify who produced the payload.
4. Whether the method works: reach a record of how the assessment method was validated.
5. How much was covered: state the population, the sample, and how it was selected.
6. How reliable that is: obtain an error characteristic that is comparable across tools and aggregable across artifacts.
7. Whether the source is reachable: resolve a reference to its source, or learn definitively that it is one-shot evidence that cannot be re-performed.
8. Whether it still stands: discover that a held artifact has been withdrawn.
Separately: an organization can attach its existing identity or attestation system without a schema change, and a consumer that doesn't understand that system can still use the core.
### Additional context
Open questions I can't answer from outside the project:
1. Is evidentiary sufficiency in scope for Gemara, or is the model deliberately limited to recording results and leaving defensibility to the consumer? A clear "no" recorded in an ADR would be a useful answer, not a bad one.
2. Items 1 and 2 are larger than the rest combined. Is there appetite, or should this issue narrow to items 3-7?
3. Does the required core warrant a `@v2` lineage or a standing allowlisted exception? The current `.oasdiff-allow` is explicitly temporary, so this means establishing a policy that doesn't exist yet.
4. Given the OpenAPI projection drops conditional constraints, how would any required core reach consumers who generate from `openapi.yaml`?
5. `gemara2oscal` currently maps only Layers 1 and 2: no `AssessmentResults`, no `Observation`. Is the measurement side unmapped because it can't be populated today, or just because it's not been done yet? OSCAL's required `observation.methods` would need a source.
6. Should the entity graph live in Gemara at all, or should Gemara reference an external inventory by stable identifier (OSCAL SSP components, a CMDB)?
Appendix G: reconciliation with existing issues and ADRs, and why this isn't #23, #170, #417, #451, or #471
### Reliability, Staleness, and Retention
| | Evidence validity periods (#451, draft) | Retention (#471) | Reliability (this) |
|---|---|---|---|
| Question | Is this still current? | How long must it be kept? | Should anyone have believed it in the first place? |
| Trigger | Time passing | A records schedule | Being challenged |
| Effect | Evidence goes stale | Evidence becomes destroyable | Evidence carries no weight |
| Scope | The single record | The single record | The record *and every conclusion above it* |
| Declared where | The plan, ahead of time | A records schedule | The measurement, at collection time |
| Consumer | Evaluator deciding whether to re-scan | Storage deciding whether it may delete | An auditor, regulator, or risk model |
| Asks of the producer | Nothing; it's plan-side | Nothing | Facts recorded at collection time that cannot be reconstructed later |
Staleness and retention are properties a consumer can evaluate at any time from data already present, and #451 declares validity *in the plan*, ahead of the measurement. Reliability depends on facts that exist only at the moment of collection: who ran it, how, against what sample, with what method. Once the artifact is written they're gone, and no later PR recovers them. Every quarter this stays open produces another quarter's worth of corpus that can't be defended retroactively.
### #23: Linkage vs. Reliability
Closed as completed by #450, because "we can now link to data that informed the conclusions drawn in the Assessments." True, and the right step. But linkage and reliability are different claims: linkage says *this conclusion cites that data*; reliability says *that data was produced by this instance, by this method, over this sample, and here's why you should believe it*. `#EvidenceMapping` gave us the first and is the right seam for the second. I'm asking to extend it, not replace it.
The issue author later confirmed they meant "specifically an in-toto attestation." Appendix F sets out why that solves authorship but can't reach the reliability question: the subject model binds by digest, and a running system has no digest.
### #170: Multi-Source Evaluation
Still open, and not a duplicate. #170 asks for aggregating logs from several tools in Layer 4/5 and deconflicting their results. The per-assessment executor gap (item 4) is a prerequisite for it: you can't deconflict across executors if no field records which executor produced which assessment, and per Appendix D you can't legitimately aggregate them either. If #170 lands first, this gets easier; it doesn't become redundant.
### #417: The `#Provenance` Type
Closed as completed. It proposed a `#Provenance` type carrying `address`, `collected`, `observed`, on `#AssessmentLog`. [ADR-0022](https://github.com/gemaraproj/website/blob/main/adrs/0022-evidence-on-assessment-log.md) rejected the separate type: "Introducing a separate type creates an artificial distinction when the relationship (opinion rooted in evidence) is the same at both layers."
That reasoning holds and I'm not reopening it. I'm asking about what the resolution didn't cover. #417 opened on the observation that "auditors cannot verify what data sources the evaluator actually consulted," and it resolved to `#Evidence` with provenance folded in. What landed answers *what was consulted*: `source: #EvidenceMapping`, with `coordinate`, `entry-id`, and `digest`. The *how* and the *by whom* were never part of the resolution, and they're the two things the reliability question turns on.
Two details from that thread worth carrying forward:
- The original proposal was explicitly modality-neutral: "works for file scans, API checks, container scans, cloud config, physical audits, and interviews." So the project already intended this to reach the testimonial end of the evidence spectrum. Nothing in the landed `#Evidence` distinguishes an interview from an API call.
- The naming exploration rejected `#Sample` because it "implies statistical sampling methodology." Worth noting for item 1, where that methodology is exactly what's being asked for, so I'd expect the naming to need care rather than reuse.
I've deliberately kept CUE out of this issue so it isn't read as a second attempt at the same type.
### #451: Evidence Validity Periods
An open draft PR, not merged; `valid-for` / `#EvidenceTypeDefinition` is not in the tree at `b47d028`, and I've described it as proposed rather than landed throughout. It changes `#AssessmentPlan.frequency` from string to int days and replaces `evidence-requirements` with an `evidence-types` array of `#EvidenceTypeDefinition` carrying validity periods, all in `policy.cue`, i.e. plan-side.
That's complementary rather than overlapping: it declares in advance how long a *kind* of evidence stays valid. It says nothing about whether a particular record was reliable when produced. See the table above, and Appendix E for what happens to it at the compatibility gate on rebase.
### #468: Payload Projection
Load-bearing here rather than cosmetic, and it's the sharpest tension in the whole issue.
Preserving verifiability means preserving raw form. A DKIM-signed email is self-authenticating (the sending domain signed the headers and body), but normalize it into a tidy payload object and that signature is destroyed. Re-encode a screenshot and any C2PA manifest goes with it. For evidence that cannot be re-performed (an email, an interview record, a one-time capture), the native verification mechanism *is* the reliability story, and it only survives verbatim.
That runs directly against ADR-0022's guidance that "it is the evaluator's responsibility to manage payload size, trim sensitive data, and ensure evidence does not cross trust boundaries inappropriately." Both instructions are correct in isolation and they conflict in practice. I think the resolution is that hard evidence must be referenced-with-digest rather than inlined, which is precisely why item 6 (resolvable references) matters more than it looks. But I'd like that resolved explicitly rather than left to each implementer.
### Adjacent Issues
- #472 (referenced evidence): directly adjacent to item 6; whatever lands there should probably carry the resolvability requirement.
- #470 / #475 (payload-or-source enforcement): `b47d028` is the commit this issue is written against, and the projection problem it exposed is open question 4.
- #74 (OSCAL relationship): the `observation.methods` question in Appendix F is a concrete, checkable claim about the converter's current output.
### Unaffected ADRs
Nothing here needs a new layer or a change to the seven-layer definitions in [ADR-0008](https://github.com/gemaraproj/website/blob/main/adrs/0008-model-lockin.md).
ADR-0023 is referenced in the `b47d028` commit message ("coordinate/entry-id comments claimed mutual exclusivity; ADR-0023... matching ADR-0022/0023") but isn't published under `adrs/`; the published set ends at 0022. I've therefore cited it by number only and haven't relied on its contents.
Related issues: #23, #74, #170, #417, #451 (draft), #468, #470 / #475, #471, #472.
Related ADRs: [ADR-0002](https://github.com/gemaraproj/website/blob/main/adrs/0002-model-schemas.md), [ADR-0008](https://github.com/gemaraproj/website/blob/main/adrs/0008-model-lockin.md), [ADR-0010](https://github.com/gemaraproj/website/blob/main/adrs/0010-dual-ladder-layers.md), [ADR-0022](https://github.com/gemaraproj/website/blob/main/adrs/0022-evidence-on-assessment-log.md).
Contributor guide
Research direction
Start by reading the cited schema sections in auditlog.cue, evaluationlog.cue, metadata.cue, collections.cue, mapping_inline.cue, entities.cue, and policy.cue, then review the linked #417 discussion and ADR-0022. Map each listed attribution, timing, sampling, validation, reference, and retraction gap to the existing schema seams; done requires an agreed scope and schema design rather than a single localized edit.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100