PolicyEngine / PolicyEngine/chronicle

Raw archive integrity: 12 manifest-declared objects were never in ledger-raw (restored); IRS re-published 2022 IRA tables; OBR blocks unattended fetch

Open
#225 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
0
Forks
1
Avg merge
11h 48m
Merged PRs (30d)
45

Description

Found while backfilling ledger-rawchronicle-raw for #143 (mechanism 3, dual-run). Every object was fetched by its manifest-recorded key and verified against the sha256 embedded in the key before being written to the new bucket. Result: 144 of the 156 manifest-declared raw objects existed in ledger-raw; 12 did not exist at all, even though their manifests record a storage.r2 URI in ledger-raw:

Package File Recovered from
census_pep/census-pep-2024-national-age-sex nc-est2024-agesex-res.csv publisher, byte-exact
census_pep/census-pep-2024-state-age-sex sc-est2024-alldata6.csv publisher, byte-exact
census_pep/census-pep-county-population-2024 co-est2024-alldata.csv publisher, byte-exact
cms_nhe/cms-nhe-historical-service-source national-health-expenditures-…-cy-1960-2024.zip publisher, byte-exact
dwp/dwp-uc-deductions-march-2025-february-2026 universal-credit-deductions-march-2025-to-february-2026.ods publisher, byte-exact
eurostat/eurostat-ilc-di01 ilc_di01.json publisher, byte-exact
eurostat/eurostat-ilc-li02 ilc_li02.json publisher, byte-exact
irs_soi/soi-county-2022 22incyallnoagi.csv publisher, byte-exact
irs_soi/soi-ira-roth-contributions-2022 22in06ira.xlsx tracked repo copy (see 2)
irs_soi/soi-ira-traditional-contributions-2022 22in05ira.xlsx tracked repo copy (see 2)
obr/obr-efo-aggregates-march-2026 efo_aggregates.xlsx tracked repo copy (see 3)
obr/obr-efo-economy-march-2026 efo_economy.xlsx tracked repo copy (see 3)

All 12 are now present in both buckets at the recorded key with the recorded hash (additive repair: nothing in ledger-raw was rewritten or moved; the recorded URIs are simply true now). A readback from chronicle-raw was hash-verified.

Three things to fix

  1. Manifests can record R2 custody that never happened. PR CI is secrets-free, so nothing checks that a storage.r2 URI resolves. Proposal: inventory-artifacts gains a --verify-r2 mode (HEAD/get by key) run by the trusted-base workflow or a scheduled job, and publish-raw writes the storage.r2 block only after a successful upload (never from a dry run).
  2. IRS re-published the 2022 IRA contribution tables. The live 22in05ira.xlsx / 22in06ira.xlsx are valid workbooks with core modified 2026-06-04 and different bytes (11,365 / 11,697 bytes vs recorded 13,932 / 13,617). The recorded bytes were restored from the tracked copies, so the fact lineage is intact, but the publisher's current release differs: register a new release revision (per the fact-identity ADR, same vintage + new bytes ⇒ new release_revision) and re-derive the IRA facts to see whether any value changed.
  3. OBR returns 403 to non-browser fetchers, so fetch-artifact cannot refresh OBR files unattended. Either a browser-like User-Agent in the fetcher (if OBR's terms allow) or a documented manual-fetch step with hash verification.

Related: #143 (bucket cutover), #221 (registration tooling).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace the inventory-artifacts, publish-raw, and fetch-artifact entry points, along with the trusted-base workflow or scheduled-job context. Review the fact-identity ADR and related issues #143 and #221 first. Done means custody verification, correct release revision handling for the IRS files, and a documented or supported path for OBR refreshes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.