apache / apache/iceberg

Proposal: ValidateTableIntegrity action

Open
#16,868 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

## Proposal

Add a ValidateTableIntegrity action that verifies the referential integrity of an Iceberg table's metadata — that every metadata, data, and delete file the table's metadata references actually exists at its stated location, or at a corresponding rewritten location when a destination is configured.

The action walks the full metadata graph — metadata.json log entries, manifest lists, manifests, data files, delete files (including V3 deletion vectors), statistics files, and partition statistics — verifying every referenced file exists at the checked location.

The action is standalone: the same interface serves a self-audit on a single table and a source-vs-destination check between two tables.

Proposal Design Doc : https://docs.google.com/document/d/13dMjNMqdumFGPl6OfdaMAomk9mWzxkXQwwjm9IA6QjU/edit?usp=sharing

Tracked across three sequential PRs:

- [ ] API contract (interface + factory + Result base)
- [ ] Spark implementation + core helpers + exception
- [ ] validate_table_integrity Spark procedure + user docs

## Two supported shapes

- Self-audit : Single table, no destination. Walks the metadata graph and verifies each referenced file exists at its stated location. Fits post-rollback verification, catalog audits, and detecting storage-side drift (files removed out-of-band while metadata still references them).

- Source-vs-destination : Walks source metadata, applies the configured prefix rewrite, and verifies each referenced file exists at the destination. Fits pre-registration copy verification, DR sync, migration audits, and any out-of-band copy audit (backups, Distcp output, manual file copies).

## Out of scope

- Content equivalence : validates file existence, not file content.
- Metadata internal consistency : the validator checks that referenced files exist, not that the metadata graph is otherwise internally consistent (e.g., manifest schema validity, snapshot log integrity, statistics correctness).

## Risk

- Low risk — additive, opt-in feature with no behavioral change for existing tables or actions.
- New action; does not modify any existing code paths.
- Read-only at the checked location — only performs file existence checks, never writes.

## Query engine

Spark

## Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start with the linked design document and the API contract stage. Then work through the planned Spark implementation, core helpers, exception, procedure, and user documentation; done means the three sequential stages are completed for both self-audit and source-vs-destination validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.