apache / apache/arrow

Parquet dictionary filter pushdown slow and large files are created

Open
#38,818 12 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

I have a sorted column of type `pa.utf8()`, about 10000 distinct values and 10M rows.
If I write it directly and set the parquet `use_dictionary`, the `read_table(..., filters=[('column', '=', 'value')])` is fast.
If I convert it to dictionary first using `pc.dictionary_encode()` and save it similarly, the same filter is 10-20x slower.

Checking the file with parquet-cli the metadata and the rowgroups are almost identical (eg. stats). However when checking the pages they differ wildly. When saving the raw strings each dict page has a dozen entries only (as expected). When saving the dictionary each dict page has every entry. I might be wrong, but I would expect only the arrow schema to have the full dict and the individual pages should have the dict fragments (which can be combined into the full dict if the whole file is read). It's not clear why the repeated dictionaries would slow down the read though, if min and max values are available in the stats.

### Component(s)

Parquet, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.