apache / apache/arrow-rs

Proposal: Add `unsafe` option to disable UTF8 validation on parquet read

Open
#6,701 1 comment 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

It is a pretty common usecase for systems (e.g. InfluxDB 3.0) to ingest data and write it as parquet files and then only ready parquet files that we wrote (and this are well formed)

Doing utf8 validation in the parquet reader take a significant amount of time for string data, likely leading to proposals to improve it like
- https://github.com/apache/arrow-rs/issues/6667

However if the usecase is to speed up parquet data that is "trusted" maybe validating at all is unecessary.

**Describe the solution you'd like**
Add an `unsafe` parquet API to disable UTF8 validation during read for users

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by tracing UTF8 validation in the parquet reader and review the linked arrow-rs issue 6667 for related performance context. Define the unsafe read API and verify that trusted reads can bypass validation while default reads retain it.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.