Proposal: ValidateTableIntegrity action
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
## Proposal
Add a ValidateTableIntegrity action that verifies the referential integrity of an Iceberg table's metadata — that every metadata, data, and delete file the table's metadata references actually exists at its stated location, or at a corresponding rewritten location when a destination is configured.
The action walks the full metadata graph — metadata.json log entries, manifest lists, manifests, data files, delete files (including V3 deletion vectors), statistics files, and partition statistics — verifying every referenced file exists at the checked location.
The action is standalone: the same interface serves a self-audit on a single table and a source-vs-destination check between two tables.
Proposal Design Doc : https://docs.google.com/document/d/13dMjNMqdumFGPl6OfdaMAomk9mWzxkXQwwjm9IA6QjU/edit?usp=sharing
Tracked across three sequential PRs:
- [ ] API contract (interface + factory + Result base)
- [ ] Spark implementation + core helpers + exception
- [ ] validate_table_integrity Spark procedure + user docs
## Two supported shapes
- Self-audit : Single table, no destination. Walks the metadata graph and verifies each referenced file exists at its stated location. Fits post-rollback verification, catalog audits, and detecting storage-side drift (files removed out-of-band while metadata still references them).
- Source-vs-destination : Walks source metadata, applies the configured prefix rewrite, and verifies each referenced file exists at the destination. Fits pre-registration copy verification, DR sync, migration audits, and any out-of-band copy audit (backups, Distcp output, manual file copies).
## Out of scope
- Content equivalence : validates file existence, not file content.
- Metadata internal consistency : the validator checks that referenced files exist, not that the metadata graph is otherwise internally consistent (e.g., manifest schema validity, snapshot log integrity, statistics correctness).
## Risk
- Low risk — additive, opt-in feature with no behavioral change for existing tables or actions.
- New action; does not modify any existing code paths.
- Read-only at the checked location — only performs file existence checks, never writes.
## Query engine
Spark
## Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Start with the linked design document and the API contract stage. Then work through the planned Spark implementation, core helpers, exception, procedure, and user documentation; done means the three sequential stages are completed for both self-audit and source-vs-destination validation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100