apache / apache/datafusion

is it possible to defer uft8 validation pass the filters

Open
#20,297 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

for clickbench query 22 - 99% of the rows did not survive the filter
Today: decode → validate 100K rows → filter → 46 rows

i wonder if it possible to decode as BinaryView → filter → validate 46 rows → cast to StringView

even it is possible looks like its a lot of work to data fusion and arrow

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the ClickBench query 22 behavior described in the issue and trace the decode, UTF-8 validation, filtering, and StringView steps in DataFusion and its Arrow integration. Define done as demonstrating whether validation can be deferred safely and documenting the required scope and performance impact; the issue names no files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.