is it possible to defer uft8 validation pass the filters
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
for clickbench query 22 - 99% of the rows did not survive the filter
Today: decode → validate 100K rows → filter → 46 rows
i wonder if it possible to decode as BinaryView → filter → validate 46 rows → cast to StringView
even it is possible looks like its a lot of work to data fusion and arrow
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by reproducing the ClickBench query 22 behavior described in the issue and trace the decode, UTF-8 validation, filtering, and StringView steps in DataFusion and its Arrow integration. Define done as demonstrating whether validation can be deferred safely and documenting the required scope and performance impact; the issue names no files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100