Proposal: Add `unsafe` option to disable UTF8 validation on parquet read
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
It is a pretty common usecase for systems (e.g. InfluxDB 3.0) to ingest data and write it as parquet files and then only ready parquet files that we wrote (and this are well formed)
Doing utf8 validation in the parquet reader take a significant amount of time for string data, likely leading to proposals to improve it like
- https://github.com/apache/arrow-rs/issues/6667
However if the usecase is to speed up parquet data that is "trusted" maybe validating at all is unecessary.
**Describe the solution you'd like**
Add an `unsafe` parquet API to disable UTF8 validation during read for users
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Research direction
Start by tracing UTF8 validation in the parquet reader and review the linked arrow-rs issue 6667 for related performance context. Define the unsafe read API and verify that trusted reads can bypass validation while default reads retain it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100