CodeForPhilly / CodeForPhilly/cfp-data-pipeline

Move DB setup into the pipeline

Open
#28 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7
Forks
3
PR merge metrics
No merged PRs in 30d

Description

From pairing w/ @chriscardillo. Right now we have a special script called `datawarehouse_setup.py` that does a little initial db setup (e.g. create a new role for the datawarehouse). It seems like we could just move this step into the pipeline. That way, adding new schemas etc.. uses the same logic as any pipeline task.

The one trick is if the db setup moves to a DAG, all other dags will need to depend on it. Chris says this is easy to do! While we're at it, we might want to enable all DAGs to be enabled at start up (vs having to manually toggle in the web ui).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.