scrapinghub / scrapinghub/dateparser

dateparse can't handle cases where the same word has different meanings

Open
#676 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Type: Bug
Dominant language
Python
Stars
2.9k
Forks
520
Avg merge
22h 56m
Merged PRs (30d)
6

Description

There are languages where some words have different meanings. This is generating some issues like this: https://github.com/scrapinghub/dateparser/issues/337

I'm creating this issue to track this and try to find a solution.

As far as I have seen, the translation is performed before the date is parsed, so we can't select the valid meaning using the other part as context. Apart from that, those words are inserted into a Python dictionary containing "word: meaning", overriding the other words with a different meaning.

Using regex simplifications could fix some use cases, but is not a valid approach for most of the cases.

In some cases (like in the word "mar" for Italian), we could detect when there are double elements (for example two months) and find if there is any duplicated key for any of them and try to find the real meaning, but this would be probably hard to implement.

Please, if you have any idea don't hesitate to comment on this issue.


List of cases:

  • In Asturian (as), Spanish (es), Filipino (fil), Friulian (fur), Galician (gl), Italian (it), Greenlandic (kl), Mauritian Creole (mfe) and Albanian (sq): “mar” means “Tuesday” and “March”.
  • The next languages use the same word for “week” and “Sunday”:
    • Embu (ebu): “kiumia”
    • Ewondo (ewo): "sɔ́ndɔ"
    • Filipino (fil) “linggo”
    • Nuosu (ii) “ꑭꆏ”
    • Georgian (ka) “კვირა”
    • Kikuyu (ki) "kiumia"
    • Bafia (ksf) “sɔ́ndǝ”
    • Indonesian (id): “minggu”
    • Ganda (lg) “sabbiiti”
    • Lao (lo) “ອາທິດ”
    • Meru (mer) “kiumia"
    • Macedonian (mk) “недела”
    • Kwasio (nmg) “sɔ́ndɔ”
    • Serbian (sr) “Недеља”
    • Yangben (yav) “sɔ́ndiɛ”

And there are a lot of other cases where the same word is used with different meanings:

  • In Bulgarian (bg): “c” means “second” and “week”.
  • In Bena (bez): “vil” means “monday” and “February”
  • In Tibetan (bo): "ཟླ་བ་" means month and monday.
  • In Breton (br): “meurzh” means “Tuesday” and “March”
  • In Welsh (cy) “maw” means “Tuesday” and “March”
  • In Taita (dav): “kaw” means “February” and “Tuesday”, “kad” means “Wednesday” and “March”, “kan” mean “April” and “Thursday” and “kas” means “Friday” and “May”.
  • In Estonian (et): “t” means “hour” and “Tuesday”, “k” means “Wednesday” and
    “month”, and “p” means “day” and “Sunday”.
  • In Gaelic (gd) “m” means “minute” and “am”
  • In Hebrew (he) "שנ׳" means “minute” and “year”.
  • In Hungarian (hu) “p” means “Friday” and “minute”.
  • In Armenian (hy) “շաբաթ” means “Saturday” and “week”
  • In Japanese (ja) “月” means “Monday” and “Month”, and "日" means “day” and “Sunday”.
  • In Cape Verdean Creole (kea) “sig” means “second” and “Monday”.
  • In Korean (ko) “월” means “Monday” and “month”, and “일” means “day” and “Sunday” .
  • In Colognian (ksh) “m” means “minute” and “month” and “s” means “second” and “hour”.
  • In Luba-Katanga (lu) “lum” means “Sunday” and “may”, and “lubingu” means “Saturday” and “week”.
  • In Makhuwa-meetto (mgh) “sab” means “July” and “Sunday”.
  • In Mundang (mua) “cla” means “February” and “Monday”.
  • In Nuer (nus) “cäŋ” means “Sunday” and “day”, and “pay” means “month” and “July”.
  • In Portuguese (pt) “seg” means “second” and “monday”.
  • In Rundi (rn) and Kinyarwanda (rw) “kan” means “Thursday” and “August”.
  • In Romanian (ro) “luni” means “month” and “Monday”, and “mar” means “Tuesday” and “March”.
  • In Samburu (saq) “ong” means “April” and “Tuesday”, “ile” means “Thursday” and “June”, and “sap” means “July” and “Friday”.
  • In Albanian (sq) “Sht” means “Saturday” and “September”.
  • In Swedish (sv) “mån” means “Monday” and “Month”, and “m” means “minute” and “month”
  • In Urdu (ur) “ہفتہ” means “Saturday” and “week”.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the linked issue and reading the translation and parsing flow described here, especially how words are stored in the Python dictionary before parsing. Done means ambiguous words retain their possible meanings and are resolved using context across the listed language cases without overriding dictionary entries.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.