scrapinghub / scrapinghub/dateparser
Timezone selection during relative/partial datetime parsing
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 520
- Avg merge
- 22h 56m
- Merged PRs (30d)
- 6
Description
My setup:
Windows 10, version 2004
Python 3.8
dateparser 1.0.0
I'm not sure if this is a code issue or a documentation issue.
From https://pypi.org/project/dateparser/ :
In case, you want to compute relative dates in UTC instead of default system’s local timezone, you can use TIMEZONE setting.
>>> parse('4 minutes ago', settings={'TIMEZONE': 'UTC'})
datetime.datetime(2017, 3, 10, 23, 27, 59, 647248, tzinfo=<StaticTzInfo 'UTC'>)
This works as described for pure relative dates such as "4 minutes ago." For datetimes without a date component, such as "7:30 PM," I thought the TIMEZONE setting might work the same way. It looks like it doesn't; instead, dateparser uses an internal UTC date and overlays the parsed time component on top of it. This is most visible when parsing a datetime in a timezone which is UTC - X when the local time is such that it's the next day in UTC. "PREFER_DATES_FROM": "future" also exacerbates the issue.
For example, if we run the following code with America/Los_Angeles as the timezone, at 5/22/2021 6:00 PM Los Angeles time (5/23/2021 1:00 AM UTC):
locale: List[str] = ["en"]
settings1: Dict[str, Any] = {
"TIMEZONE": "America/Los_Angeles"
}
date1: datetime = dateparser.parse("7:30 PM", settings=settings1, locales=locale)
print(f"Parsing with only TIMEZONE specified: {date1}")
If the datetime were parsed off America/Los_Angeles, the result should be 5/22/2021 7:30 PM. Because dateparser uses UTC internally, the actual result printed is 5/23/2021 7:30 PM.
It's possible to work around this using RELATIVE_BASE as long as the timezone is specified. RELATIVE_BASE without an explicit timezone behaves oddly instead, and can cause similar issues with date rollover. If RELATIVE_BASE is specified without a timezone, the resulting datetime appears to be parsed using the timezone specified by TIMEZONE. If RELATIVE_BASE is specified without a timezone and PREFER_DATES_FROM is "future", the resulting datetime appears to be parsed using UTC.
For example, keeping America/Los_Angeles as the timezone and running the following code at 5/22/2021 6:00 PM Los Angeles time (5/23/2021 1:00 AM UTC):
locale: List[str] = ["en"]
settings2: Dict[str, Any] = {
"TIMEZONE": "America/Los_Angeles",
"RELATIVE_BASE": datetime.now()
}
date2: datetime = dateparser.parse("7:30 PM", settings=settings2, locales=locale)
print(f"Parsing with RELATIVE_BASE and timezone not specified, and without PREFER_DATES_FROM: {date2}")
settings3: Dict[str, Any] = {
"PREFER_DATES_FROM": "future",
"TIMEZONE": "America/Los_Angeles",
"RELATIVE_BASE": datetime.now()
}
date3: datetime = dateparser.parse("7:30 PM", settings=settings3, locales=locale)
print(f"Parsing with RELATIVE_BASE and timezone not specified, and with PREFER_DATES_FROM: {date3}")
america_los_angeles_timezone: tzinfo = icu.ICUtzinfo.getInstance("America/Los_Angeles")
settings4: Dict[str, Any] = {
"PREFER_DATES_FROM": "future",
"TIMEZONE": "America/Los_Angeles",
"RELATIVE_BASE": datetime.now(tz=america_los_angeles_timezone)
}
date4: datetime = dateparser.parse("7:30 PM", settings=settings4, locales=locale)
print(f"Parsing with RELATIVE_BASE and timezone specified: {date4}")
date2 is 5/22/2021 7:30 PM. date3 is 5/23/2021 7:30 PM. date4 is the expected 5/22/2021 7:30 PM.
My confusion here stems from two points:
- The TIMEZONE setting is used for relative datetimes such as "4 minutes ago," but not partial datetimes such as "7:30 PM." If TIMEZONE is meant to work with both, that seems like a code issue to me. If it's intended behavior, I'd be appreciative if the documentation had a note to that effect, to make it clearer that TIMEZONE only works for "true" relative datetimes.
- RELATIVE_BASE seeming to change timezones based on PREFER_DATES_FROM seems odd. This seems like a code issue; if it's deliberate behavior, I'd appreciate documentation around how it works. I'd also be grateful for more documentation around RELATIVE_BASE's timezone selection; the documentation and examples all use naive datetimes, requiring trial and error to figure out how parsing works with aware datetimes. (I'm still not completely certain I understand. Unspecified RELATIVE_BASE uses UTC; specified naive RELATIVE_BASE without TIMEZONE specified seems to use the local PC timezone; specified naive RELATIVE_BASE with TIMEZONE specified and PREFER_DATES_FROM unspecified seems to use TIMEZONE; specified naive RELATIVE_BASE with TIMEZONE specified and PREFER_DATES_FROM == "future" seems to use UTC; specified aware RELATIVE_BASE seems to use RELATIVE_BASE's timezone?)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no source file or test. Start by reproducing the TIMEZONE and RELATIVE_BASE examples for partial times, then trace the parsing path that selects the base timezone. Done means the intended behavior is decided and either covered by regression tests with matching documentation or documented clearly if the behavior is intentional.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100