datafusion-contrib / datafusion-contrib/liquid-cache

Optimize filter operations on integer arrays

Open
#256 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
452
Forks
51
Avg merge
3h 10m
Merged PRs (30d)
12

Description

As @alchemist51 reported in #237, liquid cache can be slower than baseline for the following query:

SELECT SUM(backend_status_code) FROM parquet_table WHERE backend_status_code=200

The data can be found at https://github.com/XiangpengHao/liquid-cache/issues/237#issuecomment-2940875412

This query has one filter column backend_status_code and the same column is used for final projection. This is not an ideal case for filter pushdown. Nevertheless, we can still optimize it.

The problem is here: https://github.com/XiangpengHao/liquid-cache/blob/9cefcf49b970a8208a930ed8daf9bb74ecb3a0b9/src/liquid_parquet/src/liquid_array/primitive_array.rs#L202-L208

When applying a filter to the liquid primitive array, we'll first decompress the entire array (line 203), then apply filter as arrow array (line 204-205), then compress the filtered array again.

This is not optimal, a better approach is to apply the filters directly on the encoded data. In liquid format, all primitives are eventually bit-packed -- they are fixed length bit chunks. It's not too difficult to apply filters on them.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/liquid_parquet/src/liquid_array/primitive_array.rs at lines 202-208, then review the linked benchmark data and issue #237. Trace how integer primitives are encoded and how filters are applied. Done means the filter can operate on the encoded data without decompressing and recompressing the entire array, improving the shown SUM query relative to baseline.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.