Implement `DFExtensionType` for Arrow's Canonical Extension Types
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Now that it seems like we can merge https://github.com/apache/datafusion/pull/20312 soon, we should implement the full-range of Arrow's canonical extension types. Currently, only UUID is supported in the PR.
This issue tracks adding the remaining canonincal extension types:
- [x] Fixed shape tensor
- [x] Variable shape tensor
- [x] JSON
- [x] UUID
- [x] Opaque
- [x] 8-bit Boolean
- [ ] Parquet Variant
- [x] Timestamp With Offset
### Describe the solution you'd like
Implement the `DFExtensionType` similar to UUID.
The question that remains is how we implement pretty-printing for these types.
- Do we try to pretty-print tensors?
- Do we pretty-print JSON using newlines?
- I guess Parquet Variant would benefit from a nice representation in tests/CLIs. @friendlymatthew maybe you have some 2 cents here?
### Describe alternatives you've considered
We could implement the formatters within arrow-rs and just use them in DataFusion. But I am unsure where they best fit.
Maybe starting in DataFusion and migrating them to arrow-rs sometimes in the future (depending on a use case) is a good choice.
### Additional context
Some (maybe) related issues I've found:
- https://github.com/apache/datafusion/issues/7845
- https://github.com/apache/datafusion/issues/16116
Contributor guide
Research direction
Start by reviewing PR 20312 and the existing UUID implementation to understand how DFExtensionType is integrated. Then resolve the issue's open pretty-printing questions for the unchecked Parquet Variant type, with completion reflected in appropriate tests and CLI output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100