apache / apache/iceberg

Question - Is `write.target-file-size-bytes` including compression or not?

Open
#15,754 2 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Query engine

Spark, pyiceberg, or any other.

### Question

Apologies, if this is a rather stupid question, but it's currently not really clear how compression effects the targeted Parquet data file size or if it doesn't.

Scenario:
Assuming, I have an Iceberg table, without any partitioning. The total dataset is 5GBs uncompressed. I set now `write.target-file-size-bytes` to 256MB, I would expect that the resulting Parquet data files are roughly ~256MB in size on disk.
- Is my understanding so far correct?

If I set `write.parquet.compression-codec` to `zstd` and `write.parquet.compression-level` to 3 additionally, I expect the total disk size would be less than the 5GB uncompressed, because the data is now compressed.
- Will I still get ~256MB files, just each one containing more data, hence less parquet files in total?
- Or will I get smaller-sized Parquet data files on disk, but they contain roughly the same amount of data as the uncompressed 256MB parquet files from above?

For the sake of this question, I would ignore the size of manifests and metadata files.

Remark: The only thing I found so far in the documentation so far is this [Spark section](https://iceberg.apache.org/docs/latest/spark-writes/#controlling-file-sizes), but I think this one talks more about Sparks size limits and its effect on the max. parquet file size. For the sake of this question, I would ignore such limitations of the given query engine(s).

Contributor guide

Open the contributing guide

Research direction

Start with the linked Spark writes documentation section and trace the definitions of write.target-file-size-bytes, write.parquet.compression-codec, and write.parquet.compression-level across the relevant query engines mentioned. Document whether the target is compressed on-disk size or uncompressed data size, and explain the resulting file count and contents for the stated scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.