apache / apache/datafusion

Support Mapping SQL Types to the Canonical Extension Types

Open
#21,626 3 comments 0 reactions 1 assignee Claimed by @cyberbeam524 View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

After merging https://github.com/apache/datafusion/pull/21291 , we have a basic version of Arrow's canonical extension types (except Parquet Variant).

After quickly looking over the SQL AST Data Types, the following types should be considered:
- UUID: map to `arrow.uuid`
- JSON: map to `arrow.json`
- Timestamps with `TimezoneInfo::WithTimeZone`: From what I can tell, it depends on the database on how this type is interpreted. Oracle seems to retain the time zone information for each datum, while we discard the timezone and normalize it based on the configured time zone.

### Describe the solution you'd like

From what I can tell after quickly looking at the code we could change `SqlToRel::convert_simple_data_type` to retun a `Field` for the newly supported data types. Then add tests that showcase that the correct metadata is stored.

### Describe alternatives you've considered

_No response_

### Additional context

- [Oracle's TIMESTAMP WITH TIME ZONE Data Type](https://docs.oracle.com/en/database/oracle/oracle-database/12.2/sqlrf/Data-Types.html#GUID-BE23545B-469A-4A57-8D13-505F2F5DB706)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.