apache / apache/arrow-rs

`string_to_datetime` (and `Utf8` → `Timestamp(tz)` casts) fail on DST-ambiguous and nonexistent local times

Open
#11,039 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

> **Status (updated 2026-09-11):** this is a **blocker** for https://github.com/apache/datafusion/issues/25084, not a follow-up to #11037.
>
> #11038 fixes only the timestamp cast kernel. A bare string literal goes through `string_to_datetime` instead, so these DataFusion queries still fail with #11038 alone, although they are the same query to a user:
>
> ```sql
> SELECT '2024-03-10 02:30:00'::timestamp AT TIME ZONE 'America/New_York'; -- fixed by #11038
> SELECT '2024-03-10 02:30:00' AT TIME ZONE 'America/New_York'; -- needs this issue
> SET datafusion.execution.time_zone = 'America/New_York';
> SELECT '2024-03-10 02:30:00'::timestamptz; -- needs this issue
> SET datafusion.execution.time_zone = 'America/Sao_Paulo';
> SELECT '2018-11-04'::timestamptz; -- date-only site: local midnight does not exist that day
> ```
>
> The fix is #11054, stacked on #11038. It shares #11038's resolution policy (ambiguous resolves to the later instant, a gap shifts forward), so the two paths cannot drift apart.

**Describe the bug**

`string_to_datetime` in `arrow-cast/src/parse.rs`, and therefore every `Utf8`/`LargeUtf8`/`Utf8View` → `Timestamp(_, Some(tz))` cast, fails for wall-clock readings that fall on a daylight saving transition of a **named** timezone:

- the hour repeated by a "fall back" transition (ambiguous), and
- the hour skipped by a "spring forward" transition (nonexistent).

The failure is `Error parsing timestamp from '...': error computing timezone offset`. Under `CastOptions { safe: true }` the affected values silently become `NULL`. It affects both a string without a zone that is interpreted in the target timezone, and a string that carries the zone explicitly (`'2024-03-10T02:30:00 America/New_York'`).

This is the string-parsing sibling of #11037. The cause is the same: three call sites in `string_to_datetime` resolve the local time with `LocalResult::single()`, which is `None` for both `LocalResult::Ambiguous` and `LocalResult::None`:

```rust
return timezone
.from_local_datetime(&datetime)
.single()
.ok_or_else(|| err("error computing timezone offset"));
// ... (twice more, including for the parsed `Tz` suffix)
```

**To Reproduce**

arrow-cast 59.3.0, arrow-array with the `chrono-tz` feature:

```rust
use std::sync::Arc;
use arrow_array::{Array, ArrayRef, StringArray};
use arrow_array::timezone::Tz;
use arrow_cast::parse::string_to_datetime;
use arrow_cast::{cast_with_options, CastOptions};
use arrow_schema::{DataType, TimeUnit};

fn main() {
let tz: Tz = "America/New_York".parse().unwrap();
for s in [
"2024-11-01T00:00:00", // unambiguous
"2024-11-03T01:30:00", // ambiguous (fall back)
"2024-03-10T02:30:00", // nonexistent (spring forward)
"2024-03-10T02:30:00 America/New_York", // nonexistent, zone given in the string
] {
println!("{s:<40} -> {:?}", string_to_datetime(&tz, s).map(|d| d.to_rfc3339()));
}

let strings: ArrayRef = Arc::new(StringArray::from(vec![
"2024-11-01T00:00:00", "2024-11-03T01:30:00", "2024-03-10T02:30:00",
]));
let to = DataType::Timestamp(TimeUnit::Second, Some("America/New_York".into()));
let strict = CastOptions { safe: false, ..Default::default() };
println!("Utf8 -> Timestamp(tz) safe=false: {:?}", cast_with_options(&strings, &to, &strict).map(|a| a.len()));
let out = cast_with_options(&strings, &to, &CastOptions { safe: true, ..Default::default() }).unwrap();
let shown: Vec = (0..out.len())
.map(|i| arrow_cast::display::array_value_to_string(&out, i).unwrap())
.collect();
println!("Utf8 -> Timestamp(tz) safe=true: {shown:?}");
}
```

```
2024-11-01T00:00:00 -> Ok("2024-11-01T00:00:00-04:00")
2024-11-03T01:30:00 -> Err(ParseError("Error parsing timestamp from '2024-11-03T01:30:00': error computing timezone offset"))
2024-03-10T02:30:00 -> Err(ParseError("Error parsing timestamp from '2024-03-10T02:30:00': error computing timezone offset"))
2024-03-10T02:30:00 America/New_York -> Err(ParseError("Error parsing timestamp from '2024-03-10T02:30:00 America/New_York': error computing timezone offset"))
Utf8 -> Timestamp(tz) safe=false: Err(ParseError("Error parsing timestamp from '2024-11-03T01:30:00': error computing timezone offset"))
Utf8 -> Timestamp(tz) safe=true: ["2024-11-01T00:00:00-04:00", "", ""]
```

**Expected behavior**

The same resolution as #11037, which records the PostgreSQL and DuckDB outputs and links to the source of both:

- ambiguous → the **later** instant (post-transition offset): `2024-11-03T01:30:00` → `2024-11-03T01:30:00-05:00`;
- nonexistent → shift **forward** by the gap (pre-transition offset): `2024-03-10T02:30:00` → `2024-03-10T03:30:00-04:00`.

**Additional context**

- Once https://github.com/apache/arrow-rs/pull/11038 lands, arrow-cast is inconsistent with itself: casting the naive **timestamp** `2024-03-10T02:30:00` to `Timestamp(_, Some("America/New_York"))` returns `03:30:00-04:00`, while casting the **string** `'2024-03-10T02:30:00'` to the same type still errors. Same wall-clock value, same target type, different result depending on the source type.
- In DataFusion this is the difference between `'2024-03-10T02:30:00'::timestamp::timestamptz` (works after 11038) and `'2024-03-10T02:30:00'::timestamptz` (still fails) under a named session timezone. DataFusion's `timestamps.slt` currently asserts that `TIMESTAMPTZ '2023-03-12 02:00:00 America/Los_Angeles'` is an error, so that expectation flips when this is fixed.
- Kept separate from #11037 because `string_to_datetime` is public API and generic over `T: TimeZone`, so the fix has a different shape: `from_local_datetime(..).latest()` for the overlap, and for the gap a probe of `from_local_datetime(dt - 24h).earliest()` to recover the pre-transition offset followed by `from_utc_datetime`. Three call sites plus tests for the zone-suffix form.

Contributor guide

Open the contributing guide

Research direction

Start in arrow-cast/src/parse.rs at string_to_datetime and its three timezone-resolution call sites, then compare the expected policy with #11037 and #11038. Add tests covering ambiguous and nonexistent named-timezone inputs, including the zone-suffix form and safe versus strict casts. Done means ambiguous times resolve to the later instant and gaps shift forward consistently for string parsing and Utf8-family casts.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.