Rewrite data files can create more small files after partition evolution
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
When rewriting data files, the output spec defaults to the current spec. If the partition spec has evolved to be
finer-grained, the action can produce more small files instead of fewer. Like in the following example:
```java
sql("CREATE TABLE foo (id int, dt date) USING iceberg PARTITIONED BY (dt)");
sql("INSERT INTO foo VALUES (1, date('2024-01-01')), (2, date('2024-01-01'))");
sql("INSERT INTO foo VALUES (3, date('2024-01-01')), (4, date('2024-01-01'))");
// someone adds a partition field
sql("ALTER TABLE foo ADD PARTITION FIELD id");
// a background service periodically merging small files end up creating more of them
sql("CALL spark_catalog.system.rewrite_data_files("
+ "table => 'foo', strategy => 'binpack', "
+ "options => map('min-input-files', '2', 'rewrite-all', 'true'))");
```
I wonder whether we should by default write output files using the same spec as input, at least for the bin-pack strategy?
### Query engine
Spark
### Willingness to contribute
- [ ] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Assessment
This issue has not been assessed yet.