Basekick-Labs / Basekick-Labs/arc

iceberg: expired snapshots' manifests are never reclaimed, and the live manifest list grows by one DELETED-only manifest per removal pass

Open
#835 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
677
Forks
53
Avg merge
9h 14m
Merged PRs (30d)
164

Description

Found while fixing #633 (validator probes against iceberg-go v0.6.0).

## Two related growth problems in the Iceberg export metadata

**1. Orphaned manifests are never deleted.** `expireSnapshots` runs with `WithPostCommit(false)` (deliberately, #632: iceberg-go's post-commit hook would also delete Arc's data files). Nothing else deletes the manifest lists and manifests of expired snapshots, so every commit's `.avro` files stay in `/.db//metadata/` forever and are copied into every backup. Today that is one manifest list plus one small manifest per commit. After #633, a pass that re-registers a rewritten or restored path leaves behind one merged manifest the size of the table's live file list (~150-400 B per entry). Measured on a 319-file table: ~6.6 KB per ordinary commit, ~15 KB per merged commit, and the merged figure scales with the file count.

**2. The live manifest list grows with the removal history.** iceberg-go v0.6.0's snapshot producers inherit the parent's manifests verbatim, so a manifest that only holds `DELETED` entries stays in every later snapshot's manifest list. A table that adds and removes files (compaction, retention) accumulates one DELETED-only manifest per removal pass; readers open every one of them on every plan. Probe: 6 adds interleaved with 4 removes leaves 6 manifests in the current list, 4 of them DELETED-only. The #633 merge commit collapses the list when it runs, but ordinary passes do not.

## Fix shape

- A metadata-only orphan sweep in the exporter: compute the reachable set (manifest lists and manifests referenced by the snapshots of every metadata version still retained, plus the `v.metadata.json` copies) and delete other `*.avro` files under the table's metadata directory older than a grace period. Data files must never be touched.
- For the manifest-list growth, either run the merge-append (see `manifestMergeOn` in `internal/iceberg/exporter.go`) periodically when the DELETED-only manifest count exceeds a threshold, or upstream a fix to iceberg-go so producers drop manifests with no live entries (Java Iceberg's `ManifestFilterManager` behaviour).

Not a correctness issue for readers; it is unbounded disk and backup growth plus slower planning on busy tables.

Contributor guide

Open the contributing guide

Research direction

Start in internal/iceberg/exporter.go, reading expireSnapshots and manifestMergeOn. Trace how retained metadata versions reference manifest lists and manifests, then inspect the metadata directory behavior. Done means unreachable old .avro files are removed after a grace period without touching data files, and DELETED-only manifest growth is bounded or addressed through the stated merge/upstream approach.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.