arrow-py / arrow-py/arrow

.to() fails with large dates due to dateutil timestamp overflow

Open
#991 10 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
9.1k
Forks
784
PR merge metrics
No merged PRs in 30d

Description

## Issue Description
(I know the issue is actually inside dateutil, but dateutil isn't written to support larger timestamps like arrow is)

64bit
```
import arrow

date = arrow.get('3100-01-01T07:00:00Z')
print(date)
print(date.to('local'))
```
```
Traceback (most recent call last):
File "C:\Users\Matt\Desktop\test2.py", line 4, in
print(date.to("local"))
File "C:\Users\Matt\AppData\Local\Programs\Python\Python39\lib\site-packages\arrow\arrow.py", line 1076, in to
dt = self._datetime.astimezone(tz)
File "C:\Users\Matt\AppData\Local\Programs\Python\Python39\lib\site-packages\dateutil\tz\_common.py", line 144, in fromutc
return f(self, dt)
File "C:\Users\Matt\AppData\Local\Programs\Python\Python39\lib\site-packages\dateutil\tz\_common.py", line 258, in fromutc
dt_wall = self._fromutc(dt)
File "C:\Users\Matt\AppData\Local\Programs\Python\Python39\lib\site-packages\dateutil\tz\_common.py", line 222, in _fromutc
dtoff = dt.utcoffset()
File "C:\Users\Matt\AppData\Local\Programs\Python\Python39\lib\site-packages\dateutil\tz\tz.py", line 222, in utcoffset
if self._isdst(dt):
File "C:\Users\Matt\AppData\Local\Programs\Python\Python39\lib\site-packages\dateutil\tz\tz.py", line 291, in _isdst
dstval = self._naive_is_dst(dt)
File "C:\Users\Matt\AppData\Local\Programs\Python\Python39\lib\site-packages\dateutil\tz\tz.py", line 260, in _naive_is_dst
return time.localtime(timestamp + time.timezone).tm_isdst
OSError: [Errno 22] Invalid argument
```
32bit (user with a Raspberry Pi 3B+ initially reported this to me - 2050 isn't that far away)
```
import arrow

date = arrow.get('2050-01-01T07:00:00Z')
print(date)
print(date.to('local'))
```
```
File "/home/osmc/.kodi/addons/slyguy.disney.plus/resources/lib/plugin.py", line 484, in _parse_video
available = available.to('local')
File "/home/osmc/.kodi/addons/script.module.slyguy/resources/modules/arrow/arrow.py", line 722, in to
dt = self._datetime.astimezone(tz)
File "/home/osmc/.kodi/addons/script.module.slyguy/resources/modules/dateutil/tz/_common.py", line 144, in fromutc
return f(self, dt)
File "/home/osmc/.kodi/addons/script.module.slyguy/resources/modules/dateutil/tz/_common.py", line 258, in fromutc
dt_wall = self._fromutc(dt)
File "/home/osmc/.kodi/addons/script.module.slyguy/resources/modules/dateutil/tz/_common.py", line 222, in _fromutc
dtoff = dt.utcoffset()
File "/home/osmc/.kodi/addons/script.module.slyguy/resources/modules/dateutil/tz/tz.py", line 222, in utcoffset
if self._isdst(dt):
File "/home/osmc/.kodi/addons/script.module.slyguy/resources/modules/dateutil/tz/tz.py", line 291, in _isdst
dstval = self._naive_is_dst(dt)
File "/home/osmc/.kodi/addons/script.module.slyguy/resources/modules/dateutil/tz/tz.py", line 260, in _naive_is_dst
return time.localtime(timestamp + time.timezone).tm_isdst
ValueError: timestamp out of range for platform time_t
```

Due to the timestamp being too large here:
https://github.com/dateutil/dateutil/blob/master/dateutil/tz/tz.py#L259

If you hack _naive_is_dst to something like below
```
def _naive_is_dst(self, dt):
timestamp = _datetime_to_timestamp(dt) + time.timezone

MAX_TIMESTAMP = 32503719599.0
MAX_TIMESTAMP_MS = MAX_TIMESTAMP * 1000
MAX_TIMESTAMP_US = MAX_TIMESTAMP * 1000000

if timestamp > MAX_TIMESTAMP:
if timestamp < MAX_TIMESTAMP_MS:
timestamp /= 1e3
elif timestamp < MAX_TIMESTAMP_US:
timestamp /= 1e6

return time.localtime(timestamp).tm_isdst
```

it works as intended.
**So maybe arrow needs to do it's own astimezone() so it can use it's normalise timestamp function.**

For my workaround, I simply updated the dateutil code to use arrows normalize_timestamp
https://github.com/matthuisman/slyguy.addons/commit/cccc92a818ed70f30f951b15acfbf91622731c75

## System Info

- 🖥 **OS name and version**:
- 🐍 **Python version**:
- 🏹 **Arrow version**:

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the 3100 and 2050 examples, then start at Arrow.to() in arrow.py where astimezone() is called. Read the referenced dateutil/tz/tz.py _naive_is_dst path and compare it with Arrow's timestamp normalization. Done means local conversion works for these large dates without platform timestamp overflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.