How to increase average parquet file size to improve Iceberg table read performance
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Query engine
Spark in Scala. I am using iceberg-spark-runtime-3.5_2.12-1.9.2.jar.
### Question
Been dealing with pretty poor read performance on our Iceberg V2 table on Apache Spark 3.5 + Iceberg 1.9.2. Our table does have a wide nested schema (~2,000 leaf paths under struct columns), ~415 GB / ~1.29 B records.
We have `write.distribution-mode=range`, `write.format.default=parquet`, zstd compression, `write.target-file-size-bytes=250 MiB` originally. As far as I can tell, the data distribution is bimodal: a small number of hot partitions hold most rows, and a long tail of small partitions hold relatively few rows each.
Post-ingest (load + Iceberg MERGE INTO) consistently lands at:
- 38,246 files / **avg 10.86 MB** / median 457 KB / max 256 MB
- Hot partitions: well-sized (~720 files at ~255 MB each)
- Long-tail partitions: many tiny files (a partition with 6K records can hold
720 files of ~419 KB each)
I want to raise the average file size at write time without changing the partition spec, ideally at first-write time.
### What we've tried (all measured post-MERGE operation)
| # | Change | File count | Avg size | Result |
|---|--------|-----------:|---------:|--------|
| 1 | Baseline (250 MiB target, shuf=720) | 38,246 | 10.86 MB | (baseline) |
| 2 | `write.target-file-size-bytes` 250 MiB → 512 MiB | 38,110 | 10.90 MB | No change |
| 3 | `spark.sql.shuffle.partitions` 720 → 360 | 61,772 | 6.73 MB | **Worse** |
| 4 | `spark.sql.adaptive.advisoryPartitionSizeInBytes` default 64 MiB → 128 MiB | 38,110 | 10.90 MB | No change |
### Questions
1. Is there any write-side setting that materially raises the average file size at first-write time for this partition spec? Curious if anyone has achieved a higher average file size for a similar layout to our table (~7K partitions, hundreds of GB, wide nested schema).
2. If first-write is bounded, is `CALL system.rewrite_data_files` the canonical / recommended way to address this? i.e., the expected pattern is "ingest produces what it produces, then post-write compaction handles it"? If yes, is it performant to run on tables at this scale (~7K partitions, hundreds of GB)? If the post-write compaction ends up taking as long as reads/longer than reads, then maybe even a one-compaction might not be worth it.
Contributor guide
Research direction
No source files or tests are named. Start by reviewing the reported Spark 3.5 and Iceberg 1.9.2 write settings and the system.rewrite_data_files procedure, then compare behavior for the stated partition distribution; done means a documented, measured recommendation for first-write sizing and post-write compaction at this scale.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering, databases, performance
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100