PolicyEngine / PolicyEngine/chronicle
Raw archive integrity: 12 manifest-declared objects were never in ledger-raw (restored); IRS re-published 2022 IRA tables; OBR blocks unattended fetch
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 1
- Avg merge
- 11h 48m
- Merged PRs (30d)
- 45
Description
Found while backfilling ledger-raw → chronicle-raw for #143 (mechanism 3, dual-run). Every object was fetched by its manifest-recorded key and verified against the sha256 embedded in the key before being written to the new bucket. Result: 144 of the 156 manifest-declared raw objects existed in ledger-raw; 12 did not exist at all, even though their manifests record a storage.r2 URI in ledger-raw:
| Package | File | Recovered from |
|---|---|---|
census_pep/census-pep-2024-national-age-sex |
nc-est2024-agesex-res.csv | publisher, byte-exact |
census_pep/census-pep-2024-state-age-sex |
sc-est2024-alldata6.csv | publisher, byte-exact |
census_pep/census-pep-county-population-2024 |
co-est2024-alldata.csv | publisher, byte-exact |
cms_nhe/cms-nhe-historical-service-source |
national-health-expenditures-…-cy-1960-2024.zip | publisher, byte-exact |
dwp/dwp-uc-deductions-march-2025-february-2026 |
universal-credit-deductions-march-2025-to-february-2026.ods | publisher, byte-exact |
eurostat/eurostat-ilc-di01 |
ilc_di01.json | publisher, byte-exact |
eurostat/eurostat-ilc-li02 |
ilc_li02.json | publisher, byte-exact |
irs_soi/soi-county-2022 |
22incyallnoagi.csv | publisher, byte-exact |
irs_soi/soi-ira-roth-contributions-2022 |
22in06ira.xlsx | tracked repo copy (see 2) |
irs_soi/soi-ira-traditional-contributions-2022 |
22in05ira.xlsx | tracked repo copy (see 2) |
obr/obr-efo-aggregates-march-2026 |
efo_aggregates.xlsx | tracked repo copy (see 3) |
obr/obr-efo-economy-march-2026 |
efo_economy.xlsx | tracked repo copy (see 3) |
All 12 are now present in both buckets at the recorded key with the recorded hash (additive repair: nothing in ledger-raw was rewritten or moved; the recorded URIs are simply true now). A readback from chronicle-raw was hash-verified.
Three things to fix
- Manifests can record R2 custody that never happened. PR CI is secrets-free, so nothing checks that a
storage.r2URI resolves. Proposal:inventory-artifactsgains a--verify-r2mode (HEAD/get by key) run by the trusted-base workflow or a scheduled job, andpublish-rawwrites thestorage.r2block only after a successful upload (never from a dry run). - IRS re-published the 2022 IRA contribution tables. The live
22in05ira.xlsx/22in06ira.xlsxare valid workbooks with coremodified2026-06-04 and different bytes (11,365 / 11,697 bytes vs recorded 13,932 / 13,617). The recorded bytes were restored from the tracked copies, so the fact lineage is intact, but the publisher's current release differs: register a new release revision (per the fact-identity ADR, same vintage + new bytes ⇒ newrelease_revision) and re-derive the IRA facts to see whether any value changed. - OBR returns 403 to non-browser fetchers, so
fetch-artifactcannot refresh OBR files unattended. Either a browser-like User-Agent in the fetcher (if OBR's terms allow) or a documented manual-fetch step with hash verification.
Related: #143 (bucket cutover), #221 (registration tooling).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace the inventory-artifacts, publish-raw, and fetch-artifact entry points, along with the trusted-base workflow or scheduled-job context. Review the fact-identity ADR and related issues #143 and #221 first. Done means custody verification, correct release revision handling for the IRS files, and a documented or supported path for OBR refreshes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100