Error casting strings with large dates to Timestamp
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Describe the bug**
Attempting to convert a string with a valid timestamp with a large date in the ISO format (e.g., +10999-12-31T00:00:00) results in:
Error parsing timestamp from '+10999-12-31T00:00:00': error parsing date
**To Reproduce**
```rust
#[test]
fn test_cast_string_with_large_timestamp_to_timestamp() {
let array = Arc::new(StringArray::from(vec![
Some("+10999-12-31T00:00:00"),
])) as ArrayRef;
let to_type = DataType::Timestamp(TimeUnit::Second, None);
let options = CastOptions {
safe: false,
format_options: FormatOptions::default(),
};
let b = cast_with_options(&array, &to_type, &options).unwrap();
}
```
**Expected behavior**
The cast succeeds.
**Additional context**
I think this issue is similar to https://github.com/apache/arrow-rs-object-store/issues/22, but this one is for timestamps, whereas the other one affected dates.
I ran into this issue when parsing a metadata file of a delta lake table containing large dates. The table was created using Spark SQL i.e., this issue occurs in the wild.
Contributor guide
Research direction
Start with the provided Rust regression test and trace cast_with_options for StringArray to Timestamp conversion, especially the ISO timestamp parsing path. The work is done when +10999-12-31T00:00:00 casts successfully to a second-resolution timestamp and the regression test passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100