apache / apache/datafusion

Config the length of list when using In_list on parquet, rather than a const of 20.

Open
#8,609 7 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

When I use In_list Expr, if the legth of list is 19, it used 6 ms. but when the length grows to 20, it used 200ms.

### Describe the solution you'd like

in [build_predicate_expression](https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/src/physical_optimizer/pruning.rs#L775) listExpr pruning down only in `in_list.list().len() < 20`
![image](https://github.com/apache/arrow-datafusion/assets/12014068/0bc77e53-115f-43a2-9780-89fdc317077b)

I want to config the value.

### Describe alternatives you've considered

I think.
add a config in [ParquetOptions](https://github.com/apache/arrow-datafusion/blob/main/datafusion/common/src/config.rs#L281) and [ParquetExec](https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/src/datasource/physical_plan/parquet/mod.rs#L75C19-L75C19)

but I also think that is ugly, Is there a more elegant implementation?

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/core/src/physical_optimizer/pruning.rs at build_predicate_expression and trace how the in-list length threshold controls pruning. Then inspect ParquetOptions in datafusion/common/src/config.rs and ParquetExec in datafusion/core/src/datasource/physical_plan/parquet/mod.rs. Done means the threshold is configurable without losing the intended pruning behavior, with list lengths around the configured value checked for the reported performance difference.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.