scrapinghub / scrapinghub/dateparser
dateparse can't handle cases where the same word has different meanings
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 520
- Avg merge
- 22h 56m
- Merged PRs (30d)
- 6
Description
There are languages where some words have different meanings. This is generating some issues like this: https://github.com/scrapinghub/dateparser/issues/337
I'm creating this issue to track this and try to find a solution.
As far as I have seen, the translation is performed before the date is parsed, so we can't select the valid meaning using the other part as context. Apart from that, those words are inserted into a Python dictionary containing "word: meaning", overriding the other words with a different meaning.
Using regex simplifications could fix some use cases, but is not a valid approach for most of the cases.
In some cases (like in the word "mar" for Italian), we could detect when there are double elements (for example two months) and find if there is any duplicated key for any of them and try to find the real meaning, but this would be probably hard to implement.
Please, if you have any idea don't hesitate to comment on this issue.
List of cases:
- In Asturian (as), Spanish (es), Filipino (fil), Friulian (fur), Galician (gl), Italian (it), Greenlandic (kl), Mauritian Creole (mfe) and Albanian (sq): “mar” means “Tuesday” and “March”.
- The next languages use the same word for “week” and “Sunday”:
- Embu (ebu): “kiumia”
- Ewondo (ewo): "sɔ́ndɔ"
- Filipino (fil) “linggo”
- Nuosu (ii) “ꑭꆏ”
- Georgian (ka) “კვირა”
- Kikuyu (ki) "kiumia"
- Bafia (ksf) “sɔ́ndǝ”
- Indonesian (id): “minggu”
- Ganda (lg) “sabbiiti”
- Lao (lo) “ອາທິດ”
- Meru (mer) “kiumia"
- Macedonian (mk) “недела”
- Kwasio (nmg) “sɔ́ndɔ”
- Serbian (sr) “Недеља”
- Yangben (yav) “sɔ́ndiɛ”
And there are a lot of other cases where the same word is used with different meanings:
- In Bulgarian (bg): “c” means “second” and “week”.
- In Bena (bez): “vil” means “monday” and “February”
- In Tibetan (bo): "ཟླ་བ་" means month and monday.
- In Breton (br): “meurzh” means “Tuesday” and “March”
- In Welsh (cy) “maw” means “Tuesday” and “March”
- In Taita (dav): “kaw” means “February” and “Tuesday”, “kad” means “Wednesday” and “March”, “kan” mean “April” and “Thursday” and “kas” means “Friday” and “May”.
- In Estonian (et): “t” means “hour” and “Tuesday”, “k” means “Wednesday” and
“month”, and “p” means “day” and “Sunday”. - In Gaelic (gd) “m” means “minute” and “am”
- In Hebrew (he) "שנ׳" means “minute” and “year”.
- In Hungarian (hu) “p” means “Friday” and “minute”.
- In Armenian (hy) “շաբաթ” means “Saturday” and “week”
- In Japanese (ja) “月” means “Monday” and “Month”, and "日" means “day” and “Sunday”.
- In Cape Verdean Creole (kea) “sig” means “second” and “Monday”.
- In Korean (ko) “월” means “Monday” and “month”, and “일” means “day” and “Sunday” .
- In Colognian (ksh) “m” means “minute” and “month” and “s” means “second” and “hour”.
- In Luba-Katanga (lu) “lum” means “Sunday” and “may”, and “lubingu” means “Saturday” and “week”.
- In Makhuwa-meetto (mgh) “sab” means “July” and “Sunday”.
- In Mundang (mua) “cla” means “February” and “Monday”.
- In Nuer (nus) “cäŋ” means “Sunday” and “day”, and “pay” means “month” and “July”.
- In Portuguese (pt) “seg” means “second” and “monday”.
- In Rundi (rn) and Kinyarwanda (rw) “kan” means “Thursday” and “August”.
- In Romanian (ro) “luni” means “month” and “Monday”, and “mar” means “Tuesday” and “March”.
- In Samburu (saq) “ong” means “April” and “Tuesday”, “ile” means “Thursday” and “June”, and “sap” means “July” and “Friday”.
- In Albanian (sq) “Sht” means “Saturday” and “September”.
- In Swedish (sv) “mån” means “Monday” and “Month”, and “m” means “minute” and “month”
- In Urdu (ur) “ہفتہ” means “Saturday” and “week”.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the linked issue and reading the translation and parsing flow described here, especially how words are stored in the Python dictionary before parsing. Done means ambiguous words retain their possible meanings and are resolved using context across the listed language cases without overriding dictionary entries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100