apache / apache/datafusion

DataFusion support for `TimestampWithOffset`

Open
#21,116 6 comments 0 reactions 1 assignee Claimed by @coderfender View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

DataFusion has no support for the `arrow.timestamp_with_offset` which is an Arrow extension type introduced in arrow-rs 58.0.0 ([apache/arrow-rs#8743](https://github.com/apache/arrow-rs/pull/8743)). This type represents SQL `TIMESTAMP WITH TIME ZONE `by storing a UTC timestamp alongside a per row offset in minutes as a `Struct { timestamp: Timestamp(unit, "UTC"), offset_minutes: Int16 }.`

### Describe the solution you'd like

Support for this type in the existing date functions: `date_part`, `date_trunc`, `to_char`, `to_unixtime`

### Describe alternatives you've considered

The main concern I have on implementing this is about efficiency. `TimestampWithOffset` stores a per-row offset, therefore existing datetime functions cannot be applied to the array as a whole, since they require a uniform timezone for the entire array.
One approach is to just loop row by row applying each function individually, which is simple but does not take advantage of arrow's kernels. A more efficient alternative is to group rows by their `offset_minutes` value (which in theory should be low cardinality), process each group as a sub-array using the existing kernels, then reassemble the results in original row order. However, this grouping approach adds implementation complexity.

### Additional context

For substrait queries we will need substrait support to specify the offset and a different timezone to `UTC` -- https://github.com/substrait-io/substrait/issues/841

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.