Question - Is `write.target-file-size-bytes` including compression or not?
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 129
Description
### Query engine
Spark, pyiceberg, or any other.
### Question
Apologies, if this is a rather stupid question, but it's currently not really clear how compression effects the targeted Parquet data file size or if it doesn't.
Scenario:
Assuming, I have an Iceberg table, without any partitioning. The total dataset is 5GBs uncompressed. I set now `write.target-file-size-bytes` to 256MB, I would expect that the resulting Parquet data files are roughly ~256MB in size on disk.
- Is my understanding so far correct?
If I set `write.parquet.compression-codec` to `zstd` and `write.parquet.compression-level` to 3 additionally, I expect the total disk size would be less than the 5GB uncompressed, because the data is now compressed.
- Will I still get ~256MB files, just each one containing more data, hence less parquet files in total?
- Or will I get smaller-sized Parquet data files on disk, but they contain roughly the same amount of data as the uncompressed 256MB parquet files from above?
For the sake of this question, I would ignore the size of manifests and metadata files.
Remark: The only thing I found so far in the documentation so far is this [Spark section](https://iceberg.apache.org/docs/latest/spark-writes/#controlling-file-sizes), but I think this one talks more about Sparks size limits and its effect on the max. parquet file size. For the sake of this question, I would ignore such limitations of the given query engine(s).
Contributor guide
Research direction
Start with the linked Spark writes documentation section and trace the definitions of write.target-file-size-bytes, write.parquet.compression-codec, and write.parquet.compression-level across the relevant query engines mentioned. Document whether the target is compressed on-disk size or uncompressed data size, and explain the resulting file count and contents for the stated scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100