apache / apache/arrow

[C++] Temporal floor/ceil/round throws exception for timestamps ambiguous due to DST

Open
#31,443 8 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

Running pyarrow.compute.floor_temporal for timestamps that exist will throw exceptions if the times are ambiguous during the daylight savings time transitions. 

As the \*_temporal functions do not fundamentally change the times, it does not make sense that they would fail due to a timezone issue. If they must fail, it should be when the pyarrow.Timestamp is created.

 

 
```java

import pyarrow
import pyarrow.compute as pc
import datetime
import pytz

t = pyarrow.timestamp('s', tz='America/New_York')

dt = datetime.datetime(2013, 11, 3, 1, 3, 14, tzinfo = pytz.timezone('America/New_York'))

# if a timestamp must be invalid, this could fail
za = pyarrow.array([dt], t)

# raises an exception, even though this is conceptually an identity function here
pc.floor_temporal(za, unit = 'second')
```
 

And this actually works just fine (continued from above)
```java

pc.cast(   
pc.floor_temporal(       
pc.cast(za, pyarrow.timestamp('s', 'UTC')),        
unit='second'),    
pyarrow.timestamp('s','America/New_York')
)

```
 

 

 

 

**Reporter**: [Kevin Crouse](https://issues.apache.org/jira/browse/ARROW-16022)
**Watchers**: [Rok Mihevc](https://issues.apache.org/jira/browse/ARROW-16022) / @rok

**Note**: *This issue was originally created as [ARROW-16022](https://issues.apache.org/jira/browse/ARROW-16022). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Python example with pyarrow.compute.floor_temporal on the America/New_York timestamp during the DST transition, then compare it with the UTC cast path shown in the issue. Trace the temporal floor, ceil, and round implementation and its timezone validation; done means ambiguous-but-existing timestamps no longer fail for these operations, with regression coverage for the reported case.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.