scrapinghub / scrapinghub/dateparser
search_dates() raises IndexError for the zh-Hans, zh-Hant and yue locales
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 520
- Avg merge
- 22h 56m
- Merged PRs (30d)
- 6
Description
Problem
search_dates() raises IndexError: list index out of range for the zh-Hans, zh-Hant and yue locales. It is reachable through the plainest possible call, with no languages= argument at all:
>>> from dateparser.search import search_dates
>>> search_dates("聽日") # Cantonese for "tomorrow"; autodetection selects yue
IndexError: list index out of range
>>> search_dates("舊年") # "last year"
IndexError: list index out of range
It also fires whenever one of the three locales is requested explicitly, including in a mixed language list, and including for plain ASCII input:
>>> search_dates("今天", languages=["zh-Hans"]) # IndexError
>>> search_dates("2020年3月4日", languages=["zh-Hant"]) # IndexError
>>> search_dates("00Z", languages=["yue"]) # IndexError
>>> search_dates("2020年3月4日", languages=["zh-Hans", "en"]) # IndexError
>>> search_dates("3 hours ago at 5pm", languages=["zh-Hans"]) # IndexError
zh, ja and every other locale are unaffected, and dateparser.parse() does not crash (for these locales it returns None instead — that half is #1277).
Traceback:
dateparser/search/__init__.py:70 -> search/search.py:357
-> search/search.py:184
-> search/search.py:45
-> languages/locale.py:310
[original_tokens[i], original_tokens[i + 1]],
IndexError: list index out of range
Root cause
Two things combine in Locale.translate_search() (dateparser/languages/locale.py):
locale.py:277—word_joint_unsupported_languages = ["zh", "ja"]. Exactlyzhandjacarry asimplificationsblock that rewrites年/月/日into separators before this point;zh-Hans,zh-Hantandyuedo not, so they reach the word-joining branch atlocale.py:302-312with raw CJK tokens, which the other two never do.locale.py:291computesnext_worddefensively —simplified_tokens[i + 1] if i < last_token_index else ""— but the branch it guards readsoriginal_tokens[i + 1]atlocale.py:310with no equivalent bounds check. On the final token,current_and_next_joinedcan still match a dictionary entry (the joined form equals the bare word, sincenext_wordis""), the branch is taken, and the index overflows.
Introduced in c5f0fa4 (#953, 2021-08-09); git show c5f0fa4^:dateparser/languages/locale.py contains no i + 1 index at all.
Proposal
Add the missing bounds check to the condition at locale.py:302:
elif (
i < last_token_index
and current_and_next_joined in dictionary
and word not in dashes
and self.shortname not in word_joint_unsupported_languages
):
I verified this locally:
- every repro above stops raising;
- the full suite is unchanged —
24186 passed, 18 skipped, 1 xfailed, identical to baseline; search_dates("聽日")andsearch_dates("舊年")now return correct results rather than merely not crashing.
Scope and considerations
- Please do not fix this by adding the three locales to
word_joint_unsupported_languages. I tried that: it stops the crash, butsearch_dates("2020年3月4日", languages=["zh-Hans"])then returnsdatetime(6, 5, 27)— silent garbage — becausetranslate_searchyields"2020yearmarch4day". That trades a loud crash for a wrong date. - Related but distinct: #1277 (
zh-Hansvszhlocale data) and #875 (zh-Hant). Those are data gaps that makeparse()returnNone; this is a code defect that survives them. I applied #1277's proposed change in-memory andsearch_dates()still raises for今天,昨天,上个月and00Z. Sensible sequencing is the bounds guard first (crash →None, matchingparse()), the data fix second (None→ correct date). - After the guard lands,
2020年3月4日underzh-HansreturnsNonerather than a date. That is #1277's territory, not a regression from this change.
Acceptance criteria
- None of the calls listed above raises
IndexError. - Regression tests cover
zh-Hans,zh-Hantandyueintests/test_search.py, which currently contains zero references to any of the three (grep -c 'zh-Han\|yue' tests/test_search.py→0) — which is how this survived four and a half years.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in dateparser/languages/locale.py at Locale.translate_search(), especially the word-joining branch around lines 302-312, then inspect the related flow in dateparser/search/search.py. Add regression coverage in tests/test_search.py for zh-Hans, zh-Hant, and yue using the listed calls, and run those tests plus the full suite; done means none of the cases raises IndexError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100