apache / apache/arrow-rs

Error casting strings with large dates to Timestamp

Open
#7,208 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Describe the bug**

Attempting to convert a string with a valid timestamp with a large date in the ISO format (e.g., +10999-12-31T00:00:00) results in:

Error parsing timestamp from '+10999-12-31T00:00:00': error parsing date

**To Reproduce**

```rust
#[test]
fn test_cast_string_with_large_timestamp_to_timestamp() {
let array = Arc::new(StringArray::from(vec![
Some("+10999-12-31T00:00:00"),
])) as ArrayRef;
let to_type = DataType::Timestamp(TimeUnit::Second, None);
let options = CastOptions {
safe: false,
format_options: FormatOptions::default(),
};
let b = cast_with_options(&array, &to_type, &options).unwrap();
}
```

**Expected behavior**
The cast succeeds.

**Additional context**

I think this issue is similar to https://github.com/apache/arrow-rs-object-store/issues/22, but this one is for timestamps, whereas the other one affected dates.

I ran into this issue when parsing a metadata file of a delta lake table containing large dates. The table was created using Spark SQL i.e., this issue occurs in the wild.

Contributor guide

Open the contributing guide

Research direction

Start with the provided Rust regression test and trace cast_with_options for StringArray to Timestamp conversion, especially the ISO timestamp parsing path. The work is done when +10999-12-31T00:00:00 casts successfully to a second-resolution timestamp and the regression test passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.