Config the length of list when using In_list on parquet, rather than a const of 20.
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
When I use In_list Expr, if the legth of list is 19, it used 6 ms. but when the length grows to 20, it used 200ms.
### Describe the solution you'd like
in [build_predicate_expression](https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/src/physical_optimizer/pruning.rs#L775) listExpr pruning down only in `in_list.list().len() < 20`

I want to config the value.
### Describe alternatives you've considered
I think.
add a config in [ParquetOptions](https://github.com/apache/arrow-datafusion/blob/main/datafusion/common/src/config.rs#L281) and [ParquetExec](https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/src/datasource/physical_plan/parquet/mod.rs#L75C19-L75C19)
but I also think that is ugly, Is there a more elegant implementation?
### Additional context
_No response_
Contributor guide
Research direction
Start in datafusion/core/src/physical_optimizer/pruning.rs at build_predicate_expression and trace how the in-list length threshold controls pruning. Then inspect ParquetOptions in datafusion/common/src/config.rs and ParquetExec in datafusion/core/src/datasource/physical_plan/parquet/mod.rs. Done means the threshold is configurable without losing the intended pruning behavior, with list lengths around the configured value checked for the reported performance difference.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100