DeleteOrphanFiles can corrupt other tables that share the same table location
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.4.3
### Query engine
Spark
### Bug description
`DeleteOrphanFiles` deletes files belonging to *other* tables when two Iceberg tables share the same `LOCATION`. It computes "reachable files" from only the target table's metadata, then recursively lists the whole location and deletes everything else — including the second table's data and metadata files.
This happens easily in practice: a test table is created by copying `SHOW CREATE TABLE` of a production table but only the name is changed, forgetting to change `LOCATION`. We hit this in production (irreversible loss on the production table) and found 100+ such shared-location pairs in our environment.
issue #4265 was closed via #4652 but the same-location scenario was never actually covered, (PR #4652) only catches scheme/authority mismatches, not the dominant shared-path case.
## Reproduction
```sql
-- Recently written files (within the last 24 hours) are removed only when spark.testing is enabled.
spark-sql --conf spark.testing=true
-- Create test db
CREATE DATABASE IF NOT EXISTS spark_catalog.repro_db_v2;
-- Table A at a fixed location
CREATE TABLE spark_catalog.repro_db_v2.a (id BIGINT) USING iceberg LOCATION 'hdfs://HDFS78000003/tmp/iceberg_shared_repro_v2';
INSERT INTO spark_catalog.repro_db_v2.a VALUES (1L), (2L);
-- Table B accidentally (or temporarily) points at the SAME location
CREATE TABLE spark_catalog.repro_db_v2.b (id BIGINT) USING iceberg LOCATION 'hdfs://HDFS78000003/tmp/iceberg_shared_repro_v2';
INSERT INTO spark_catalog.repro_db_v2.b VALUES (3L);
-- Run orphan cleanup on A
CALL spark_catalog.system.remove_orphan_files(
table => 'repro_db_v2.a',
older_than => TIMESTAMP '2099-09-10 12:00:00.000');
-- Result: B's metadata.json / manifest / data files under the shared
-- directory are now deleted -> B is corrupted / unloadable.
SELECT * from spark_catalog.repro_db_v2.b;
```
### Proposed fix
Add a default-on guard to `DeleteOrphanFiles`:
1. List `metadata/*.metadata.json` (+ `.metadata.json.gz`) in `table.location()/metadata`.
2. For each file not in the current table's known version set, read its `table-uuid`.
- Different uuid → another table shares the location → abort (default).
- Unreadable (legacy/compressed/corrupt) → treat as suspicious → abort (fail-safe).
- Same uuid → older version of this table → skip.
3. Only proceed when no foreign uuid is found.
The logic lives in `iceberg-core` (reusable utility) and is wired into `DeleteOrphanFilesSparkAction`, so other engines can reuse it. Default = abort; an explicit opt-out allows forced cleanup when a shared location is intentional.
### Ask
Happy to contribute a PR.
Contributor guide
Research direction
Start with DeleteOrphanFiles and DeleteOrphanFilesSparkAction in iceberg-core and the Spark action wiring. Reproduce the shared-location case from the SQL example, then trace how metadata files and table UUIDs are discovered. Done means cleanup no longer deletes another table's files by default, while the documented opt-out still permits intentional shared-location cleanup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100