apache / apache/amoro

[Bug]: IcebergTableMaintainer orphan file cleanup can corrupt other tables sharing the same location

Open
#4,292 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
1.2k
Forks
395
Avg merge
4d 10h
Merged PRs (30d)
33

Description

### What happened?

## Bug description

Amoro's `IcebergTableMaintainer.cleanOrphanFiles()` can delete files belonging to *other* Iceberg tables when two tables share the same `LOCATION`.

It computes reachable files from only the target table's metadata, then recursively lists the whole location and deletes everything else — including a second table's data, manifest, and metadata files under the same directory.

This can be triggered inadvertently in practice: for example, a table is created by copying `SHOW CREATE TABLE` of another table but forgetting to change `LOCATION`. We hit this in production (irreversible data loss) and found 100+ shared-location pairs in our environment.

## Proposed fix

Add a default-on guard in `IcebergTableMaintainer` before orphan cleanup:

1. List `metadata/*.metadata.json` (+ `.metadata.json.gz`) under `table.location()/metadata`.
2. For each file not in the table's own version set, read its `table-uuid`:
- Different uuid → another table shares the location → **skip cleanup** (default).
- Unreadable (legacy/compressed/corrupt) → treat as suspicious → skip (fail-safe).
- Same uuid → older version of this table → ignore.
3. Only proceed when no foreign uuid is found.

Add table property `clean-orphan-file.ignore-location-conflict` (default `false`) to opt out when a shared location is intentional.

### Affects Versions

all

### What table formats are you seeing the problem on?

Iceberg

### What engines are you seeing the problem on?

AMS

### How to reproduce

1. Create Iceberg table A at `hdfs://ns/tmp/shared`.
2. Create Iceberg table B at the **same** location `hdfs://ns/tmp/shared`.
3. Configure orphan file cleanup and trigger it on A:
- Set catalog-level table property `clean-orphan-file.enabled=true` to enable periodic cleanup.
- Set `clean-orphan-file.min-existing-time-minutes=2` so orphan files are eligible for deletion within minutes (instead of the default 2 days), for quick observation.
4. Restart AMS so the configuration takes effect, and let the scheduled orphan file cleanup task run.
5. B's files under the shared directory are deleted → B becomes corrupted/unloadable.

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Research direction

Start with IcebergTableMaintainer.cleanOrphanFiles() and trace how reachable files and the recursive location listing are computed. Reproduce the shared-location case with two Iceberg tables, then verify cleanup skips when metadata reveals a foreign table UUID, while the default-off conflict property permits intentional sharing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.