apache / apache/arrow

[Python] [Parquet] Compression degradation when column type changed from INT64 to INT32

Open
#35,726 22 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Within a CSV of ~17M rows, I have a column of unique integers that are fairly uniformly distributed between 0 and 200,000,000. I am reading the CSV as follows:

```
from pyarrow import csv, parquet

def file_to_data_frame_to_parquet(local_file: str, parquet_file: str) -> None:
table = csv.read_csv(local_file, convert_options=csv.ConvertOptions(strings_can_be_null=True))
parquet.write_table(table, parquet_file, compression='zstd')
```

When I read the column without any type specification, the uncompressed size is 133.1MB, and the compressed size is 18.0 MB.

When I add an explicit type mapping for that column in the `read_csv` step of either `uint32` or `int32`, the total uncompressed size shrinks to 67.0 MB, but the compressed size expands to 55.8 MB. (I'm getting these statistics from the parquet schema metadata functions in DuckDB, but I've validated the difference is real from the total file size.)

This degradation stays the same with a variety of different changes to settings/envs:
- pyarrow 11.0.0 and 12.0.0
- MacOS 13.4 and Ubuntu 20.04
- ZSTD and GZIP compression (GZIP performs better than ZSTD but the degradation is still there)
- explicitly expanding row groups/write batches to 1GB
- Versions 1.0, 2.4, 2.6, data page versions 1.0 and 2.0
- Dictionaries enabled/disabled
- Dictionary page sizes of 1KB, 1MB, 1GB
- When the table is sorted sequentially by the column or when it is sorted at random

### Component(s)

Parquet, Python

Contributor guide

Open the contributing guide

Research direction

Start with the reported Python reproduction using pyarrow.csv.read_csv and pyarrow.parquet.write_table, comparing inferred INT64 output with explicit uint32 or int32 output under the listed compression settings. Trace the Parquet encoding and compression path to identify why the narrower type produces larger compressed data; done means the regression is explained and the explicit-type case no longer shows the reported degradation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.