Infer JSON fields of arrays of `u8` as `ByteArray`
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I am generating sample JSON data using a tool that does not support parquet. I am trying to import the JSON into a parquet file using the inference logic. I found that this does not work when I want a byte array column to be generated because the inference does not support it.
**Describe the solution you'd like**
JSON arrays of natural numbers that fit into u8 are inferred as byte array type instead of array
**Describe alternatives you've considered**
Change JSON -> Parquet generation tool to force the column schema to be byte array without relying on inference. Alternatively, add special support for hexadecimal strings (but this sounds even more hacky as such strings are not standardized in JSON).
**Additional context**
Contributor guide
Research direction
Start by locating the JSON inference logic and the path that converts inferred schemas to Parquet. Trace how arrays of natural numbers are classified, then verify that values fitting u8 are represented as ByteArray rather than array; add or run focused inference coverage if the project provides it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100