scrapinghub / scrapinghub/dateparser

ISO 8601 YYYYMMDD format

Open
#939 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.9k
Forks
520
Avg merge
22h 56m
Merged PRs (30d)
6

Description

First, thanks for the great project!

Our project standardized on using dates in the ISO 8601 YYYYMMDD format. According to wikipedia:

the standard allows both the "YYYY-MM-DD" and YYYYMMDD formats for complete calendar date representations

Dates in this format return None in the following example:

from datetime import datetime

import dateparser

result = dateparser.parse("20210629", languages=["en"])  # returns None
assert result == datetime(2021, 6, 29)  # fails

I then tried the following, which produced incorrect results.

from datetime import datetime

from dateparser.date import DateDataParser
from dateparser_data.settings import default_parsers

parsers = default_parsers.copy()
parsers.append('no-spaces-time')

ddp = DateDataParser(languages=["en"], settings={"PARSERS": parsers})

result = ddp.get_date_data("20210629")  # returns datetime(1062, 2, 2, 9, 0)
assert result["date_obj"] == datetime(2021, 6, 29)  # fails

This issue may be more or less related to a handful of other issues as well: #360, #765, #867, #914, some of which PR #790 may help address, although that may not address this particular format.

Since I began typing this issue, I figured out that I am able to get the correct results with dateparser.parse("20210629", date_formats=["%Y%m%d"]), which might work in this case. Although I do wonder, should this format be expected to work without explicitly defining the date_format?

On a related note, from what I could tell, the DateDataParser does not accept a date_formats parameter, even in the settings dictionary, instead it looks like that must be passed through to the method called to parse the string, which seemed a bit unintuitive to me.

Also, while I haven't yet tried using this at scale, I wonder if the dateparser.parse function might have some opportunity for performance improvement by not instantiating the DateDataParser class on each call, one possible (untested) approach might be something like the following:

from functools import lru_cache

@lru_cache
def get_ddp(**kwargs):
    return DateDataParser(**kwargs)

I would be happy to open separate issues for the above points if preferred.

Sorry for the lengthy post, and thanks again for the your contributions to the community.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the YYYYMMDD case through dateparser.parse and DateDataParser.get_date_data, comparing the default parser list with an explicit date_formats=["%Y%m%d"] format. Read the handling of the no-spaces-time parser and DateDataParser settings. Done means the documented ISO 8601 compact date parses correctly without producing incorrect dates, with the scope of the separate API and performance questions clarified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.