apache / apache/arrow-rs

[parquet_derive] Add inner struct and enum support for writing and reading

Open
#7,734 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Basically I need to be able to write large vectors of data to/from parquet files. And my record struct type has an enum in it with fields. I would like to be able to do this by just adding `#[derive(ParquetRecordWriter, ParquetRecordReader)]` into my types. This should also support structs within structs, or enums within structs, etc.

It would also be nice if I didn't have to hardcode the number of records to be read. It should be able to know when to stop.

**Additional context**

My development journal can be found [here]() for my problem context.

Contributor guide

Open the contributing guide

Research direction

Start by reading the parquet_derive implementations behind ParquetRecordWriter and ParquetRecordReader. Determine how nested structs and enums, including enums with fields, should map to Parquet data and how reading can detect the end without a hardcoded record count. Done means these derive macros support the requested nested types and automatic stopping behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.