apache / apache/arrow-rs

Infer JSON fields of arrays of `u8` as `ByteArray`

Open
#1,619 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I am generating sample JSON data using a tool that does not support parquet. I am trying to import the JSON into a parquet file using the inference logic. I found that this does not work when I want a byte array column to be generated because the inference does not support it.

**Describe the solution you'd like**
JSON arrays of natural numbers that fit into u8 are inferred as byte array type instead of array

**Describe alternatives you've considered**
Change JSON -> Parquet generation tool to force the column schema to be byte array without relying on inference. Alternatively, add special support for hexadecimal strings (but this sounds even more hacky as such strings are not standardized in JSON).

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by locating the JSON inference logic and the path that converts inferred schemas to Parquet. Trace how arrays of natural numbers are classified, then verify that values fitting u8 are represented as ByteArray rather than array; add or run focused inference coverage if the project provides it.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.