apache / apache/arrow

[C++] strptime rolls-over dates not in range for current month

Open
#31,374 7 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Component: R Priority: Critical Status: needs champion Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

I noticed some potentially unexpected behaviour when converting from string to date. Days that are out of bounds for the given month are rolled over into the following month.

I think the expected behaviour would be to either error (Python) or return NULL/NA (R), but not to roll over dates in the following month.
```r

library(arrow, warn.conflicts = FALSE)
library(lubridate, warn.conflicts = FALSE)
library(dplyr, warn.conflicts = FALSE)

df <- tibble::tibble(string_date = "1999-02-30")

# base R returns NA
df %>%
mutate(date = strptime(string_date, format = "%Y-%m-%d"))
#> # A tibble: 1 × 2
#> string_date date
#>
#> 1 1999-02-30 NA

# arrow rolls over the 30th of February into the 2nd of March
df %>%
arrow_table() %>%
mutate(date = strptime(string_date, format = "%Y-%m-%d")) %>%
collect()
#> # A tibble: 1 × 2
#> string_date date
#>
#> 1 1999-02-30 1999-03-02 00:00:00
```
Thanks Alenka, Joris and Rok for helping me with the Python examples:
pandas:
```python

>>> import pandas as pd
>>> pd.to_datetime("1999-02-30", format="%Y-%m-%d")
...
ValueError: time data 1999-02-30 doesn't match format specified
```
datetime:
```python

>>> import datetime
>>> from datetime import datetime
>>> datetime.strptime("1999-02-30", "%Y-%m-%d")
...
ValueError: day is out of range for month
```
arrow:
```python

>>> import pyarrow.compute as pc
>>> print(pc.strptime("1999-02-30", format="%Y-%m-%d", unit="s"))
1999-03-02 00:00:00
```

**Reporter**: [Dragoș Moldovan-Grünfeld](https://issues.apache.org/jira/browse/ARROW-15948) / @dragosmg
**Watchers**: [Rok Mihevc](https://issues.apache.org/jira/browse/ARROW-15948) / @rok
#### Related issues:
- [[C++] Add error handling option to StrptimeOptions](https://github.com/apache/arrow/issues/20115) (is related to)

**Note**: *This issue was originally created as [ARROW-15948](https://issues.apache.org/jira/browse/ARROW-15948). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Reproduce the invalid-date case through pyarrow.compute.strptime using "1999-02-30" and format "%Y-%m-%d". Then read the C++ strptime implementation and related issue #20115 about StrptimeOptions, establish the intended invalid-date behavior, and add regression coverage showing that out-of-range days no longer silently roll into the next month.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.