scrapinghub / scrapinghub/dateparser

Aren't "zh" (Chinese) and "zh-Hans" (Chinese, traditional script) the same language?

Open
#1,277 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.9k
Forks
520
Avg merge
22h 56m
Merged PRs (30d)
6

Description

Hi,

While working on another project, we saw that parsing Chinese dates rarely works when the language zh-Hans, whereas it works fine with zh.

Example output:

Detecting text: 2024年6月1日
ja: 2024-06-01 00:00:00
zh: 2024-06-01 00:00:00
zh-Hans-HK: None
zh-Hans: None
zh-Hant-HK: None
zh-Hant: None

From what I understand about CLDR, Chinese (simplified) is the default locale for Chinese: https://st.unicode.org/cldr-apps/v#/zh_Hans//

In the dateparser source code, zh-Hans appears to be a subset of zh, sharing most translation tables but lacking simplification rules, which seems to cause the parsing failures.

I made a proposed change here: https://github.com/scrapinghub/dateparser/compare/master...Merinorus:dateparser:chinese-dates

Before submitting a PR, I’d like to confirm whether my understanding is correct.

Thank you!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the Chinese date examples for zh, zh-Hans, zh-Hans-HK, zh-Hant, and zh-Hant-HK. Read the dateparser source around the zh-Hans translation tables and simplification rules, then compare the proposed chinese-dates change; done means the affected Chinese locales parse the demonstrated date consistently and tests cover the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
internationalization
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.