datafusion-contrib / datafusion-contrib/liquid-cache
Optimize filter operations on integer arrays
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 452
- Forks
- 51
- Avg merge
- 3h 10m
- Merged PRs (30d)
- 12
Description
As @alchemist51 reported in #237, liquid cache can be slower than baseline for the following query:
SELECT SUM(backend_status_code) FROM parquet_table WHERE backend_status_code=200
The data can be found at https://github.com/XiangpengHao/liquid-cache/issues/237#issuecomment-2940875412
This query has one filter column backend_status_code and the same column is used for final projection. This is not an ideal case for filter pushdown. Nevertheless, we can still optimize it.
The problem is here: https://github.com/XiangpengHao/liquid-cache/blob/9cefcf49b970a8208a930ed8daf9bb74ecb3a0b9/src/liquid_parquet/src/liquid_array/primitive_array.rs#L202-L208
When applying a filter to the liquid primitive array, we'll first decompress the entire array (line 203), then apply filter as arrow array (line 204-205), then compress the filtered array again.
This is not optimal, a better approach is to apply the filters directly on the encoded data. In liquid format, all primitives are eventually bit-packed -- they are fixed length bit chunks. It's not too difficult to apply filters on them.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/liquid_parquet/src/liquid_array/primitive_array.rs at lines 202-208, then review the linked benchmark data and issue #237. Trace how integer primitives are encoded and how filters are applied. Done means the filter can operate on the encoded data without decompressing and recompressing the entire array, improving the shown SUM query relative to baseline.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100