scrapinghub / scrapinghub/dateparser

Dateparsed hangs minutes on certain dates

Open
#1,185 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.9k
Forks
520
Avg merge
22h 56m
Merged PRs (30d)
6

Description

I don't have much details atm but I noticed that in certain cases the performance is awful. I wouldn't say it's a specific version, it has been always like that though rare. I took an eye on it.

E.g.

188.09s call     manhattan_hub/helpers/tests/test_time_helper.py::test_extract_datetime_from[: 20/05/20-2020-05-20-mmdd]

@pytest.mark.parametrize("ocr_date, parsed, date_format", ocr_to_iso)
def test_extract_datetime_from(ocr_date, parsed, date_format):
    with freeze_time("2021-12-02"):
        assert (
            str(
                time_helper.extract_datetime_from(
                    ocr_date,
                    date_format=date_format,
                    ignore_dates_older_than_years=10,
                    max_future_date=time_helper.x_days_in_future(12 * 31),
                )
            ).split(" ")[0]
            == parsed
        )

The tests are launched in parallel, the code deployed in Django environment with multiple wsgi workers.

Python 3.7.13
dateparser 1.1.8

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the slow case in manhattan_hub/helpers/tests/test_time_helper.py::test_extract_datetime_from, using the 20/05/20 input and the shown date format and date limits. Trace time_helper.extract_datetime_from and dateparser behavior to identify why this case takes minutes, then add regression coverage showing the case completes promptly without changing the expected parsed date.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.