scrapinghub / scrapinghub/dateparser

search_dates() raises IndexError for the zh-Hans, zh-Hant and yue locales

Open Beginner friendly
#1,372 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

search_dates Status: Bug confirmed Type: Bug
Dominant language
Python
Stars
2.9k
Forks
520
Avg merge
22h 56m
Merged PRs (30d)
6

Description

Problem

search_dates() raises IndexError: list index out of range for the zh-Hans, zh-Hant and yue locales. It is reachable through the plainest possible call, with no languages= argument at all:

>>> from dateparser.search import search_dates
>>> search_dates("聽日")          # Cantonese for "tomorrow"; autodetection selects yue
IndexError: list index out of range
>>> search_dates("舊年")          # "last year"
IndexError: list index out of range

It also fires whenever one of the three locales is requested explicitly, including in a mixed language list, and including for plain ASCII input:

>>> search_dates("今天", languages=["zh-Hans"])                 # IndexError
>>> search_dates("2020年3月4日", languages=["zh-Hant"])          # IndexError
>>> search_dates("00Z", languages=["yue"])                      # IndexError
>>> search_dates("2020年3月4日", languages=["zh-Hans", "en"])    # IndexError
>>> search_dates("3 hours ago at 5pm", languages=["zh-Hans"])   # IndexError

zh, ja and every other locale are unaffected, and dateparser.parse() does not crash (for these locales it returns None instead — that half is #1277).

Traceback:

dateparser/search/__init__.py:70  -> search/search.py:357
                                  -> search/search.py:184
                                  -> search/search.py:45
                                  -> languages/locale.py:310
      [original_tokens[i], original_tokens[i + 1]],
IndexError: list index out of range

Root cause

Two things combine in Locale.translate_search() (dateparser/languages/locale.py):

  1. locale.py:277word_joint_unsupported_languages = ["zh", "ja"]. Exactly zh and ja carry a simplifications block that rewrites // into separators before this point; zh-Hans, zh-Hant and yue do not, so they reach the word-joining branch at locale.py:302-312 with raw CJK tokens, which the other two never do.
  2. locale.py:291 computes next_word defensively — simplified_tokens[i + 1] if i < last_token_index else "" — but the branch it guards reads original_tokens[i + 1] at locale.py:310 with no equivalent bounds check. On the final token, current_and_next_joined can still match a dictionary entry (the joined form equals the bare word, since next_word is ""), the branch is taken, and the index overflows.

Introduced in c5f0fa4 (#953, 2021-08-09); git show c5f0fa4^:dateparser/languages/locale.py contains no i + 1 index at all.

Proposal

Add the missing bounds check to the condition at locale.py:302:

                elif (
                    i < last_token_index
                    and current_and_next_joined in dictionary
                    and word not in dashes
                    and self.shortname not in word_joint_unsupported_languages
                ):

I verified this locally:

  • every repro above stops raising;
  • the full suite is unchanged — 24186 passed, 18 skipped, 1 xfailed, identical to baseline;
  • search_dates("聽日") and search_dates("舊年") now return correct results rather than merely not crashing.

Scope and considerations

  • Please do not fix this by adding the three locales to word_joint_unsupported_languages. I tried that: it stops the crash, but search_dates("2020年3月4日", languages=["zh-Hans"]) then returns datetime(6, 5, 27) — silent garbage — because translate_search yields "2020yearmarch4day". That trades a loud crash for a wrong date.
  • Related but distinct: #1277 (zh-Hans vs zh locale data) and #875 (zh-Hant). Those are data gaps that make parse() return None; this is a code defect that survives them. I applied #1277's proposed change in-memory and search_dates() still raises for 今天, 昨天, 上个月 and 00Z. Sensible sequencing is the bounds guard first (crash → None, matching parse()), the data fix second (None → correct date).
  • After the guard lands, 2020年3月4日 under zh-Hans returns None rather than a date. That is #1277's territory, not a regression from this change.

Acceptance criteria

  • None of the calls listed above raises IndexError.
  • Regression tests cover zh-Hans, zh-Hant and yue in tests/test_search.py, which currently contains zero references to any of the three (grep -c 'zh-Han\|yue' tests/test_search.py0) — which is how this survived four and a half years.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in dateparser/languages/locale.py at Locale.translate_search(), especially the word-joining branch around lines 302-312, then inspect the related flow in dateparser/search/search.py. Add regression coverage in tests/test_search.py for zh-Hans, zh-Hant, and yue using the listed calls, and run those tests plus the full suite; done means none of the cases raises IndexError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.