scrapinghub / scrapinghub/dateparser
Aren't "zh" (Chinese) and "zh-Hans" (Chinese, traditional script) the same language?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 520
- Avg merge
- 22h 56m
- Merged PRs (30d)
- 6
Description
Hi,
While working on another project, we saw that parsing Chinese dates rarely works when the language zh-Hans, whereas it works fine with zh.
Example output:
Detecting text: 2024年6月1日
ja: 2024-06-01 00:00:00
zh: 2024-06-01 00:00:00
zh-Hans-HK: None
zh-Hans: None
zh-Hant-HK: None
zh-Hant: None
From what I understand about CLDR, Chinese (simplified) is the default locale for Chinese: https://st.unicode.org/cldr-apps/v#/zh_Hans//
In the dateparser source code, zh-Hans appears to be a subset of zh, sharing most translation tables but lacking simplification rules, which seems to cause the parsing failures.
I made a proposed change here: https://github.com/scrapinghub/dateparser/compare/master...Merinorus:dateparser:chinese-dates
Before submitting a PR, I’d like to confirm whether my understanding is correct.
Thank you!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the Chinese date examples for zh, zh-Hans, zh-Hans-HK, zh-Hant, and zh-Hant-HK. Read the dateparser source around the zh-Hans translation tables and simplification rules, then compare the proposed chinese-dates change; done means the affected Chinese locales parse the demonstrated date consistently and tests cover the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- internationalization
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100