apache / apache/arrow-rs

[parquet_derive] support decoding `Option<T>` for `ParquetRecordReader`

Open
#6,879 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I'm trying to have a single struct with optional fields that can be written and read as parquet. I want to have something like this:
```
#[derive(ParquetRecordWriter, ParquetRecordReader, Default)]
pub(crate) struct TabularIdentifier {
id: u64,
type: String,
maybe_thing: Option,
}
```

This however does not compile. I get a compiler error that looks like this:
```
error: proc-macro derive panicked
--> src/bin/commands/generators/parquet/records.rs:146:31
|
146 | #[derive(ParquetRecordWriter, ParquetRecordReader, Default)]
| ^^^^^^^^^^^^^^^^^^^
|
= help: message: not implemented: Unsupported: Option(
TypePath(
Type::Path {
qself: None,
path: Path {
leading_colon: None,
segments: [
PathSegment {
ident: Ident {
ident: "String",
span: #0 bytes(90325..90331),
},
arguments: PathArguments::None,
},
],
},
},
),
)

```

**Describe the solution you'd like**
It would be really helpful to be able to encode and decode a single struct that has optional fields via `parquet-derive`.

**Describe alternatives you've considered**
An alternative might be a more manual process that reads out the parquet data and constructs the struct. That is a fair bit more manual than I would like.

Contributor guide

Open the contributing guide

Research direction

Start in the parquet-derive implementation of the ParquetRecordReader derive, using the reported Option panic and the TabularIdentifier example as entry points. Trace how supported field types are decoded, then verify that a struct with Option fields can be read and written as parquet without the derive panic.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.