Double slash (//) in table location path
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.10.1 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
### Problem
When creating Iceberg tables through Spark with S3 location, the table location contains double slashes (`//`) in the path. This behavior occurs regardless of whether the table is partitioned or not. The double slash in the path causes the `OPTIMIZE` operation to behave incorrectly and sometimes delete files, leading to table corruption.
### Example
```sql
CREATE TABLE test.test.test (
test_rk integer
)
WITH (
compression_codec = 'ZSTD',
format = 'PARQUET',
format_version = 2,
location = 's3a://test//test_8262bea6c787' -- ⚠️ Double slash after 'test/'
)
```
### Example 2
```sql
CREATE TABLE test.test.test (
test_rk decimal(21, 0),
test real,
test_l varchar,
test_stat real,
test_id integer
)
WITH (
compression_codec = 'ZSTD',
format = 'PARQUET',
format_version = 2,
location = 's3a://test//test',
partitioning = ARRAY['test_id']
)
```
Component | Version
-- | --
iceberg-spark-runtime | 3.5_2.12-1.10.1
iceberg-aws-bundle | 1.10.1
Spark | 3.5.6
Filesystem | S3A
simple spark query: `df.format('iceberg').mode('overwrite').saveAsTable("test.test.test")`
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start by reproducing the table creation with the shown Spark saveAsTable call and S3A locations, then trace how Iceberg records and normalizes the table location before OPTIMIZE. Done means the resulting location does not contain the unintended double slash and OPTIMIZE no longer deletes files or corrupts the table.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, hadoop, java
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100