apache / apache/arrow-rs

Propose change timestamp casting with timezone to without timezones (also parsing of timestamps without timezones)

Open
#5,827 15 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
This is in the context of implementing `date_bin` for timestamps with timezones: https://github.com/apache/datafusion/issues/10602

I made https://github.com/apache/arrow-rs/pull/5826 to document the behavior of casting timestamps and I found it very confusing. Specifically when you cast from `Timestamp(None)` to `Timestamp(Some(tz))` and then back to `Timetamp(None)` the underlying timestamp values are changed as shown in this example

```rust
use arrow_array::Int64Array;
use arrow_array::types::{TimestampSecondType};
use arrow_cast::{cast, display};
use arrow_array::cast::AsArray;
use arrow_schema::{DataType, TimeUnit};
let data_type = DataType::Timestamp(TimeUnit::Second, None);
let data_type_tz = DataType::Timestamp(TimeUnit::Second, Some("-05:00".into()));
let a = Int64Array::from(vec![1_000_000_000, 2_000_000_000, 3_000_000_000]);
let b = cast(&a, &data_type).unwrap(); // cast to timestamp without timezone
let b = b.as_primitive::(); // downcast to result type
assert_eq!(2_000_000_000, b.value(1)); // values are still the same

// Convert timestamps without a timezone to timestamps with a timezone
let c = cast(&b, &data_type_tz).unwrap();
let c = c.as_primitive::(); // downcast to result type
assert_eq!(2_000_018_000, c.value(1)); // value has been adjusted by offset

// Convert from timestamp with timezone back to timestamp without timezone
let d = cast(&c, &data_type).unwrap();
let d = d.as_primitive::(); // downcast to result type
assert_eq!(2_000_018_000, d.value(1)); // <---- **** THIS VALUE IS DIFFERENT THAN IT WAS INITITALLY
assert_eq!("2033-05-18T08:33:20", display::array_value_to_string(&d, 1).unwrap());
```

Thus I wanted to discuss if we should change the behavior to make it less surprising or if there was a reason to leave the current behavior

**Describe the solution you'd like**

I propose making `casting timestamp with a timezone to timestamp without a timezone` do the inverse of `casting timestamp withpit a timezone to timestamp with a timezone`

This would mean the final value of d in the above example is `2_000_000_000`, not `2_000_018_000`

**Describe alternatives you've considered**
Leave existing behavior

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start with the timestamp casting behavior shown through arrow_cast::cast, arrow_cast::display, and arrow_schema::DataType and TimeUnit. Review the discussion in Apache DataFusion issue 10602 and the behavior documented by arrow-rs pull request 5826 before deciding whether the proposed inverse conversion is appropriate. Done means the casting and timestamp-parsing behavior has a documented decision and corresponding implementation or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.