Basekick-Labs / Basekick-Labs/arc
iceberg: expired snapshots' manifests are never reclaimed, and the live manifest list grows by one DELETED-only manifest per removal pass
- Dominant language
- Go
- Stars
- 677
- Forks
- 53
- Avg merge
- 9h 14m
- Merged PRs (30d)
- 164
Description
Found while fixing #633 (validator probes against iceberg-go v0.6.0).
## Two related growth problems in the Iceberg export metadata
**1. Orphaned manifests are never deleted.** `expireSnapshots` runs with `WithPostCommit(false)` (deliberately, #632: iceberg-go's post-commit hook would also delete Arc's data files). Nothing else deletes the manifest lists and manifests of expired snapshots, so every commit's `.avro` files stay in `/.db//metadata/` forever and are copied into every backup. Today that is one manifest list plus one small manifest per commit. After #633, a pass that re-registers a rewritten or restored path leaves behind one merged manifest the size of the table's live file list (~150-400 B per entry). Measured on a 319-file table: ~6.6 KB per ordinary commit, ~15 KB per merged commit, and the merged figure scales with the file count.
**2. The live manifest list grows with the removal history.** iceberg-go v0.6.0's snapshot producers inherit the parent's manifests verbatim, so a manifest that only holds `DELETED` entries stays in every later snapshot's manifest list. A table that adds and removes files (compaction, retention) accumulates one DELETED-only manifest per removal pass; readers open every one of them on every plan. Probe: 6 adds interleaved with 4 removes leaves 6 manifests in the current list, 4 of them DELETED-only. The #633 merge commit collapses the list when it runs, but ordinary passes do not.
## Fix shape
- A metadata-only orphan sweep in the exporter: compute the reachable set (manifest lists and manifests referenced by the snapshots of every metadata version still retained, plus the `v.metadata.json` copies) and delete other `*.avro` files under the table's metadata directory older than a grace period. Data files must never be touched.
- For the manifest-list growth, either run the merge-append (see `manifestMergeOn` in `internal/iceberg/exporter.go`) periodically when the DELETED-only manifest count exceeds a threshold, or upstream a fix to iceberg-go so producers drop manifests with no live entries (Java Iceberg's `ManifestFilterManager` behaviour).
Not a correctness issue for readers; it is unbounded disk and backup growth plus slower planning on busy tables.
Contributor guide
Research direction
Start in internal/iceberg/exporter.go, reading expireSnapshots and manifestMergeOn. Trace how retained metadata versions reference manifest lists and manifests, then inspect the metadata directory behavior. Done means unreachable old .avro files are removed after a grace period without touching data files, and DELETED-only manifest growth is bounded or addressed through the stated merge/upstream approach.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100