apache / apache/arrow-rs

Extension type <--> Parquet LogicalType registry / user defined mappings

Open
#8,479 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 16h
Merged PRs (30d)
168

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

The Parquet type system includes LogicalTypes types without a direct arrow equivalent, such as JSON, Variant, and UUID

However, Arrow includes the idea of "Extension" types that add extra semantics to an existing Arrow physical type, and the arrow-rs parquet reader will automatically map these the relevant parquet types to a canonical Arrow extension type if the `arrow_canonical_extension_types` feature is set.

However, right now that mapping of Parquet LogicalType --> Arrow (Canonical) ExtensionType is hard coded, which is unfortunate as it means:
1. Users can not override the mapping (if they want to write their own implementation of parquet LogicalTypes, for example)
2. The code has a bunch of `#[cfg(...)]` sprinkled in it -- see https://github.com/apache/arrow-rs/pull/8409 for an example

**Describe the solution you'd like**
@paleolimbot suggested on https://github.com/apache/arrow-rs/pull/8409/files#r2371071848 that we could maintain some sort of registry that was more ergonomic to configure and would allow user defined extension types

**Describe alternatives you've considered**
Quoting @paleolimbot on https://github.com/apache/arrow-rs/pull/8409/files#r2371071848:

> you could also consider an injection approach like:

```rust
pub trait ParquetArrowExtension {
fn try_from_logical_type(&self, mut arrow_field: Field, logical_type: &LogicalType) -> Result>;
fn try_to_logical_type(&self, &Field) -> Result>;
}
```

...and maintain a registry of those in the reader/writer options. Then you don't need compile time flags to support the extensions (something like DataFusion or a derivative could wire it all together at runtime).

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the hard-coded Parquet LogicalType-to-Arrow extension mappings and the discussion in pull request 8409. Compare the proposed ParquetArrowExtension trait with reader and writer options, then define the registry behavior and user-defined mapping lifecycle. Done means mappings are configurable without the current compile-time flags and existing canonical mappings still work.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.