Parquet dictionary filter pushdown slow and large files are created
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
I have a sorted column of type `pa.utf8()`, about 10000 distinct values and 10M rows.
If I write it directly and set the parquet `use_dictionary`, the `read_table(..., filters=[('column', '=', 'value')])` is fast.
If I convert it to dictionary first using `pc.dictionary_encode()` and save it similarly, the same filter is 10-20x slower.
Checking the file with parquet-cli the metadata and the rowgroups are almost identical (eg. stats). However when checking the pages they differ wildly. When saving the raw strings each dict page has a dozen entries only (as expected). When saving the dictionary each dict page has every entry. I might be wrong, but I would expect only the arrow schema to have the full dict and the individual pages should have the dict fragments (which can be combined into the full dict if the whole file is read). It's not clear why the repeated dictionaries would slow down the read though, if min and max values are available in the stats.
### Component(s)
Parquet, Python
Contributor guide
Assessment
This issue has not been assessed yet.