apache / apache/iceberg

Rewrite data files can create more small files after partition evolution

Open
#15,465 3 comments 1 reaction 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

When rewriting data files, the output spec defaults to the current spec. If the partition spec has evolved to be
finer-grained, the action can produce more small files instead of fewer. Like in the following example:
```java
sql("CREATE TABLE foo (id int, dt date) USING iceberg PARTITIONED BY (dt)");
sql("INSERT INTO foo VALUES (1, date('2024-01-01')), (2, date('2024-01-01'))");
sql("INSERT INTO foo VALUES (3, date('2024-01-01')), (4, date('2024-01-01'))");

// someone adds a partition field
sql("ALTER TABLE foo ADD PARTITION FIELD id");

// a background service periodically merging small files end up creating more of them
sql("CALL spark_catalog.system.rewrite_data_files("
+ "table => 'foo', strategy => 'binpack', "
+ "options => map('min-input-files', '2', 'rewrite-all', 'true'))");
```
I wonder whether we should by default write output files using the same spec as input, at least for the bin-pack strategy?

### Query engine

Spark

### Willingness to contribute

- [ ] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.