eclipse-score / eclipse-score/score

Standardize Evidence Artifacts and Schema

Open
#2,856 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Starlark
Stars
109
Forks
105
Avg merge
1d 9h
Merged PRs (30d)
21

Description

## Goal
Standardize the artifact set emitted by all onboarded repositories.

## Why
Developers and governance tooling need consistent machine-readable evidence across repos.

## Required Artifact Contract

**Base fields (all repos)**
- `execution_lane` (`oss_default` | `accelerated`)
- `policy_decision` (allow | deny, with machine-readable reason)
- `test_summary`
- `quality_or_traceability_metrics`
- `dependency_manifest_summary`
- `license_policy_result`
- `reproducibility_metadata` (rerun id, tool versions, environment hash)

**Extended fields for docs-as-code / assurance domain**
- `impacted_elements` — list of need IDs or argument element IDs directly impacted by the change
- `impact_class` per element — one of: `direct_recheck` | `indirect_propagation` | `revision_required`
- `consistency_rule_ids` — which rules triggered the impact classification
- `affected_artifact_type` — `requirement` | `guideline` | `test_link` | `process_requirement` | `evidence`
- `change_scenario_id` — reference to the specific change scenario class that was matched
- `responsible_role` — who is accountable for the outcome (default: `pr_creator`)
- `escalation_role` — who to escalate failures to (default: `harness_maintainer`)
- `waiver_authority` — who can approve waivers (default: `release_approver`)

**Provenance fields** (all repos):
- `execution_timestamp` — ISO 8601 timestamp
- `python_version` — interpreter version
- `environment_hash` — stable hash of execution environment
- `gate_script_version` — validator version

These extended fields enable the harness outer loop to grep structured traces and identify recurring failure patterns without reading raw sphinx-build output.

## Queryability Rules

- Artifacts must be machine-readable (`json`, `jsonl`, or schema-validated yaml)
- Filenames must be stable and consistent across runs so simple tools such as `grep` and `cat` work reliably
- Raw stdout/stderr may be retained as optional debug artifacts, but never as the primary evidence contract
- A small summary index (`evolution_summary.jsonl` or equivalent) must exist so agents start from the index before drilling into traces
- The docs-as-code pilot should standardize at least `gate_output.json`, `impacted_elements.json`, and `score.json` per task

## Tasks
- Define artifact manifest and schema versions.
- Reuse/align with existing docs-as-code schema patterns.
- Validate schema compatibility in CI.
- Document retention and discoverability.
- Define compatibility policy for schema evolution.
- Define canonical filenames and directory layout for per-run and per-task trace artifacts.

## Docs-as-code pilot tasks
- Extend current metrics artifacts with required provenance fields.
- Validate artifact schemas in CI before policy decision.
- Ensure accelerated-lane artifacts remain verifiable by OSS validators.
- Add `evolution_summary.jsonl` or equivalent summary index for selective navigation of prior runs.
- Document the canonical per-task trace layout under `runs///traces//`.

## Done When
- Pilot repos emit the same artifact contract.
- CI rejects malformed or incomplete artifact payloads.
- Schema version policy is documented and enforced.
- Agents can locate the relevant trace files for one failure class without reading raw command output dumps.

Parent: eclipse-score/score#2852

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.