scrapinghub / scrapinghub/dateparser
Update CLDR data
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 520
- Avg merge
- 22h 56m
- Merged PRs (30d)
- 6
Description
Once we merge this: https://github.com/scrapinghub/dateparser/pull/825
we will be able to upgrade the CLDR data easily. By doing it we will improve this library as we will be able to add support for more locales, fix old language bugs, etc.
I will try to explain how to do it.
- Increase the
cldr_versionnumber (indateparser_scripts.utils.get_raw_data). We should go version by version to avoid too much files to check. - Run
python dateparser_scripts/get_cldr_data.py. This will download the new JSON files. - Check the new files. If there are new locales supported (new files or new regions), we should add them to the docs and probably add tests. We can add some languages to the
avoid_languageslists or remove some from it if the language is now fully supported. - Run
python dateparser_scripts/write_complete_data.py. This will create the new.pyfiles by merging thejsonfiles with theyamlfiles. - Run
python dateparser_scripts/order_languages.pyto sort them and update thelanguages_info.py - Run the tests (
tox). - If tests fail it could be because some old valid words have been removed from the new files. In this case, we can change/remove the test or add the old words to the
yamlfiles. - Run
dateparser_scripts/update_supported_languages_and_locales.pyto update the supported locales in the docs.
Next versions to update:
- 32.0.0 (Nov 2017)
- 33.0.0 (Mar 2018)
- 34.0.0 (Oct 2018)
- 35.0.0 (Mar 2019)
- 35.1.0 (Apr 2019)
- 36.0.0 (Oct 2019)
- 37.0.0 (Oct 2020)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review PR #825, then inspect the cldr_version setting in dateparser_scripts.utils and run the listed CLDR data scripts in order. Check generated JSON and Python files, update locale documentation and tests as needed, run tox, and finish by updating supported languages and locales.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, localization, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 42/100