apache / apache/datafusion

Implement `DFExtensionType` for Arrow's Canonical Extension Types

Open
#21,144 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Now that it seems like we can merge https://github.com/apache/datafusion/pull/20312 soon, we should implement the full-range of Arrow's canonical extension types. Currently, only UUID is supported in the PR.

This issue tracks adding the remaining canonincal extension types:
- [x] Fixed shape tensor
- [x] Variable shape tensor
- [x] JSON
- [x] UUID
- [x] Opaque
- [x] 8-bit Boolean
- [ ] Parquet Variant
- [x] Timestamp With Offset

### Describe the solution you'd like

Implement the `DFExtensionType` similar to UUID.

The question that remains is how we implement pretty-printing for these types.
- Do we try to pretty-print tensors?
- Do we pretty-print JSON using newlines?
- I guess Parquet Variant would benefit from a nice representation in tests/CLIs. @friendlymatthew maybe you have some 2 cents here?

### Describe alternatives you've considered

We could implement the formatters within arrow-rs and just use them in DataFusion. But I am unsure where they best fit.

Maybe starting in DataFusion and migrating them to arrow-rs sometimes in the future (depending on a use case) is a good choice.

### Additional context

Some (maybe) related issues I've found:
- https://github.com/apache/datafusion/issues/7845
- https://github.com/apache/datafusion/issues/16116

Contributor guide

Open the contributing guide

Research direction

Start by reviewing PR 20312 and the existing UUID implementation to understand how DFExtensionType is integrated. Then resolve the issue's open pretty-printing questions for the unchecked Parquet Variant type, with completion reflected in appropriate tests and CLI output.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.