devinit / devinit/ddw-analyst-ui

Refactor iati_transactions code into a general-purpose reload function that could create multiple tables

Open
#772 2 comments 0 reactions 0 assignees View on GitHub
ETL humportal pinned
Dominant language
TypeScript
Stars
2
Forks
2
Avg merge
4d 27m
Merged PRs (30d)
7

Description

At present we have the following data update scripts:

1. iati.sh
2. iati_datastore.sh
3. iati_registry_refresh.sh
4. iati_transactions.sh
5. iati_transactions_retry.sh

We should clean up, simplify, and refactor these scripts such that they:

1. Run Python/iati_refresh.py, marking IATI datasets as either new, modified, or stale in the iati_registry_metadata table.
2. Run a new script based on Python/iati_transactions.py, which is capable of modularly modifying multiple IATI-based tables at the moment new data is loaded (IATI nomenclature would call this iati_reload.py).
3. Run auxiliary scripts such as Python/iati_rhfp.py which rely on the entire data structure and don't need to be run strictly on new/modified files.

The idea is that instead of running iati_transactions.sh run once, at which point the information about whether a dataset is new is destroyed, any number of tables can be progressively built during the reload process just as iati_transactions.py is.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading iati.sh, iati_datastore.sh, iati_registry_refresh.sh, iati_transactions.sh, iati_transactions_retry.sh, and the referenced Python scripts iati_refresh.py, iati_transactions.py, and iati_rhfp.py. Map their current responsibilities and define the reload flow so registry status remains available while multiple tables are built; done means the scripts are simplified and the new modular reload process supports the listed auxiliary work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, shell
Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.