devinit / devinit/ddw-analyst-ui
Refactor iati_transactions code into a general-purpose reload function that could create multiple tables
- Dominant language
- TypeScript
- Stars
- 2
- Forks
- 2
- Avg merge
- 4d 27m
- Merged PRs (30d)
- 7
Description
At present we have the following data update scripts:
1. iati.sh
2. iati_datastore.sh
3. iati_registry_refresh.sh
4. iati_transactions.sh
5. iati_transactions_retry.sh
We should clean up, simplify, and refactor these scripts such that they:
1. Run Python/iati_refresh.py, marking IATI datasets as either new, modified, or stale in the iati_registry_metadata table.
2. Run a new script based on Python/iati_transactions.py, which is capable of modularly modifying multiple IATI-based tables at the moment new data is loaded (IATI nomenclature would call this iati_reload.py).
3. Run auxiliary scripts such as Python/iati_rhfp.py which rely on the entire data structure and don't need to be run strictly on new/modified files.
The idea is that instead of running iati_transactions.sh run once, at which point the information about whether a dataset is new is destroyed, any number of tables can be progressively built during the reload process just as iati_transactions.py is.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading iati.sh, iati_datastore.sh, iati_registry_refresh.sh, iati_transactions.sh, iati_transactions_retry.sh, and the referenced Python scripts iati_refresh.py, iati_transactions.py, and iati_rhfp.py. Map their current responsibilities and define the reload flow so registry status remains available while multiple tables are built; done means the scripts are simplified and the new modular reload process supports the listed auxiliary work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, shell
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100