apache / apache/iceberg

FileAlreadyExistsException in CopyTable with Position Delete Files

Open
#14,589 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.5.0

### Query engine

Spark

### Please describe the bug 🐞

Problem
When copying Iceberg tables with position delete files using SparkActions.copyTable(), the operation fails with FileAlreadyExistsException during parallel processing:
org.apache.hadoop.fs.FileAlreadyExistsException: /staging/00001-deletes.parquet already exists

This issue is reproducible when:
Multiple manifests reference the same position delete file (e.g., after manifest compaction)
Different position delete files have the same filename in different directories

Root Cause
The stagingPath() method only used the filename to generate staging paths:

// OLD CODE
private static String stagingPath(String originalPath, String stagingLocation) {
return stagingLocation + fileName(originalPath); // Only filename!
}

Collision scenarios:
dir1/00001-deletes.parquet → staging/00001-deletes.parquet
dir2/00001-deletes.parquet → staging/00001-deletes.parquet ❌ COLLISION
When Spark processes manifests in parallel (mapPartitions at line 784), multiple tasks simultaneously try to write to the same staging path, causing the exception.

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start at stagingPath() and the mapPartitions call at line 784; compare how original paths are converted into staging paths when manifests are processed in parallel. Reproduce the listed duplicate-reference and same-filename cases, then verify that copyTable() completes without FileAlreadyExistsException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.