NYCPlanning / NYCPlanning/data-engineering
PLUTO: On-Demand QA
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43
- Forks
- 3
- Avg merge
- 23h 3m
- Merged PRs (30d)
- 44
Description
As discussed in Roadmapping, we believe that there's value in generalizing the QA comparison code between builds to be done on-the-fly, with builds of a dataset specified outside of a given build process. An impactful first step would be to implement such a process for two builds of PLUTO.
Rough Acceptance Criteria
- Add a github action to generate dataset comparisons, with the following inputs: product name, product build 1, product build 2
- The action should kick off a job that will download the specified build of PLUTO to the runner, import them to postgres (probably edm-data. But just be sure to delete the tables afterward) and invoke the existing QA script to generate the outputs.
- The outputs should be uploaded a new s3 bucket,
edm-dataset-comparisons, under a folder with the product name, then into a folder for the comparison itself (still somewhat tbd). Given two sample build names,ar-23v1andar-24v1, the full folder structure might look something like this:
edm-dataset-comparisons/ar-23v1/ar-24v1/qaqc_mismatch.csv - Bonus points: import the data into Postgres, into a table with columns for the dataset, build 1 name, build 2 name, and the comparison itself as JSON. This would make for painless integration with the QA app.
For inspiration, here's a Draft PR with a data model, and further explanations.
cc @damonmcc @fvankrieken @sf-dcp FYI, I don't have immediate plans to get started (though I will, if time permits) so don't hold back if you'd like to take this on.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the existing QA script and the Draft PR 524 for its data model and explanations. Then map the GitHub Action inputs to downloading two PLUTO builds, importing them into Postgres, generating comparison outputs, and uploading them to the edm-dataset-comparisons S3 bucket; done means the comparison workflow runs and cleans up its tables.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, github-actions, postgres, python
- Domain
- ci-cd, cloud, data, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100