CodeForPhilly / CodeForPhilly/cfp-data-pipeline
Move DB setup into the pipeline
- Dominant language
- Python
- Stars
- 7
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
From pairing w/ @chriscardillo. Right now we have a special script called `datawarehouse_setup.py` that does a little initial db setup (e.g. create a new role for the datawarehouse). It seems like we could just move this step into the pipeline. That way, adding new schemas etc.. uses the same logic as any pipeline task.
The one trick is if the db setup moves to a DAG, all other dags will need to depend on it. Chris says this is easy to do! While we're at it, we might want to enable all DAGs to be enabled at start up (vs having to manually toggle in the web ui).
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.