scrapinghub / scrapinghub/dateparser
ISO 8601 YYYYMMDD format
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 520
- Avg merge
- 22h 56m
- Merged PRs (30d)
- 6
Description
First, thanks for the great project!
Our project standardized on using dates in the ISO 8601 YYYYMMDD format. According to wikipedia:
the standard allows both the "YYYY-MM-DD" and YYYYMMDD formats for complete calendar date representations
Dates in this format return None in the following example:
from datetime import datetime
import dateparser
result = dateparser.parse("20210629", languages=["en"]) # returns None
assert result == datetime(2021, 6, 29) # fails
I then tried the following, which produced incorrect results.
from datetime import datetime
from dateparser.date import DateDataParser
from dateparser_data.settings import default_parsers
parsers = default_parsers.copy()
parsers.append('no-spaces-time')
ddp = DateDataParser(languages=["en"], settings={"PARSERS": parsers})
result = ddp.get_date_data("20210629") # returns datetime(1062, 2, 2, 9, 0)
assert result["date_obj"] == datetime(2021, 6, 29) # fails
This issue may be more or less related to a handful of other issues as well: #360, #765, #867, #914, some of which PR #790 may help address, although that may not address this particular format.
Since I began typing this issue, I figured out that I am able to get the correct results with dateparser.parse("20210629", date_formats=["%Y%m%d"]), which might work in this case. Although I do wonder, should this format be expected to work without explicitly defining the date_format?
On a related note, from what I could tell, the DateDataParser does not accept a date_formats parameter, even in the settings dictionary, instead it looks like that must be passed through to the method called to parse the string, which seemed a bit unintuitive to me.
Also, while I haven't yet tried using this at scale, I wonder if the dateparser.parse function might have some opportunity for performance improvement by not instantiating the DateDataParser class on each call, one possible (untested) approach might be something like the following:
from functools import lru_cache
@lru_cache
def get_ddp(**kwargs):
return DateDataParser(**kwargs)
I would be happy to open separate issues for the above points if preferred.
Sorry for the lengthy post, and thanks again for the your contributions to the community.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the YYYYMMDD case through dateparser.parse and DateDataParser.get_date_data, comparing the default parser list with an explicit date_formats=["%Y%m%d"] format. Read the handling of the no-spaces-time parser and DateDataParser settings. Done means the documented ISO 8601 compact date parses correctly without producing incorrect dates, with the scope of the separate API and performance questions clarified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100