Let arrow schemas specify the ENUM type from parquet schemas
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 168
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I am exporting data from an application using an Arrow schema - the data is then written into a parquet file. Some fields are logically enums, and I would like to export them as the ENUM type in Parquet, but there is currently no way to specify that.
**Describe the solution you'd like**
In parquet, ENUM is an annotation of a BYTE_ARRAY, similar to STRING. Unlike String, it's parametrized with the allowable values.
It sounds like it might be an extension type, as in #5822 .
Contributor guide
Research direction
Review the requested Arrow-to-Parquet schema behavior and the extension-type discussion in issue #5822 first. Define how allowable enum values should be represented and how the resulting Parquet ENUM annotation should be verified; the issue is done when Arrow schemas can express this information and it is preserved when writing Parquet.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100