Proposal: optional audit manifest for benchmark result provenance

Proposal

Would this project be open to a small, dependency-light result audit manifest for CLIP benchmark runs?

The goal is to make benchmark outputs easier to review, compare, and cite safely, especially when results are reused in public reports, model cards, or downstream eval dashboards.

Suggested manifest shape

A run could optionally emit a sidecar JSON file like:

{
  "schema_version": "clip_benchmark.audit_manifest.v1",
  "model": "...",
  "dataset": "...",
  "task": "zeroshot_classification",
  "metrics_path": "...",
  "provenance": {
    "benchmark_repo": "LAION-AI/CLIP_benchmark",
    "benchmark_commit": "...",
    "created_at": "..."
  },
  "risk_labels": ["dataset_bias", "multimodal_safety"],
  "claim_status": "diagnostic",
  "evidence_refs": [
    {
      "source_id": "...",
      "kind": "benchmark_result",
      "redaction_status": "safe_for_public_log"
    }
  ]
}

Why this may be useful

• makes it clearer which run produced which public claim
• separates diagnostic/internal runs from reportable results
• preserves provenance such as benchmark commit, dataset, task, and result path
• gives downstream users a place to attach risk labels or audit notes without changing core metrics
• helps prevent accidental overclaiming when benchmark outputs are copied into model cards or reports

Scope I would keep small

If maintainers think this is useful, I can prepare a PR that:

• adds an optional manifest writer
• keeps the schema small and documented
• does not change existing benchmark metrics
• does not add any external service dependency
• includes a short README/example and a smoke test

This is motivated by work in the AANA project around audit-safe AI evaluation artifacts, but the contribution would be generic to CLIP_benchmark and would not require AANA as a dependency.

Start by locating how benchmark runs currently emit or record result files and where a README example could fit. Define the optional manifest's small documented schema, add a writer without changing existing metrics, and verify it with the proposed smoke test.