[parquet_derive] support decoding `Option<T>` for `ParquetRecordReader`
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I'm trying to have a single struct with optional fields that can be written and read as parquet. I want to have something like this:
```
#[derive(ParquetRecordWriter, ParquetRecordReader, Default)]
pub(crate) struct TabularIdentifier {
id: u64,
type: String,
maybe_thing: Option,
}
```
This however does not compile. I get a compiler error that looks like this:
```
error: proc-macro derive panicked
--> src/bin/commands/generators/parquet/records.rs:146:31
|
146 | #[derive(ParquetRecordWriter, ParquetRecordReader, Default)]
| ^^^^^^^^^^^^^^^^^^^
|
= help: message: not implemented: Unsupported: Option(
TypePath(
Type::Path {
qself: None,
path: Path {
leading_colon: None,
segments: [
PathSegment {
ident: Ident {
ident: "String",
span: #0 bytes(90325..90331),
},
arguments: PathArguments::None,
},
],
},
},
),
)
```
**Describe the solution you'd like**
It would be really helpful to be able to encode and decode a single struct that has optional fields via `parquet-derive`.
**Describe alternatives you've considered**
An alternative might be a more manual process that reads out the parquet data and constructs the struct. That is a fair bit more manual than I would like.
Contributor guide
Research direction
Start in the parquet-derive implementation of the ParquetRecordReader derive, using the reported Option panic and the TabularIdentifier example as entry points. Trace how supported field types are decoded, then verify that a struct with Option fields can be read and written as parquet without the derive panic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100