apache / apache/arrow-rs

Let arrow schemas specify the ENUM type from parquet schemas

Open
#7,050 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 16h
Merged PRs (30d)
168

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

I am exporting data from an application using an Arrow schema - the data is then written into a parquet file. Some fields are logically enums, and I would like to export them as the ENUM type in Parquet, but there is currently no way to specify that.

**Describe the solution you'd like**

In parquet, ENUM is an annotation of a BYTE_ARRAY, similar to STRING. Unlike String, it's parametrized with the allowable values.

It sounds like it might be an extension type, as in #5822 .

Contributor guide

Open the contributing guide

Research direction

Review the requested Arrow-to-Parquet schema behavior and the extension-type discussion in issue #5822 first. Define how allowable enum values should be represented and how the resulting Parquet ENUM annotation should be verified; the issue is done when Arrow schemas can express this information and it is preserved when writing Parquet.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.