NYCPlanning / NYCPlanning/data-engineering

PLUTO: On-Demand QA

Open
#532 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

db-pluto
Dominant language
Python
Stars
43
Forks
3
Avg merge
23h 3m
Merged PRs (30d)
44

Description

As discussed in Roadmapping, we believe that there's value in generalizing the QA comparison code between builds to be done on-the-fly, with builds of a dataset specified outside of a given build process. An impactful first step would be to implement such a process for two builds of PLUTO.

Rough Acceptance Criteria
  • Add a github action to generate dataset comparisons, with the following inputs: product name, product build 1, product build 2
  • The action should kick off a job that will download the specified build of PLUTO to the runner, import them to postgres (probably edm-data. But just be sure to delete the tables afterward) and invoke the existing QA script to generate the outputs.
  • The outputs should be uploaded a new s3 bucket, edm-dataset-comparisons, under a folder with the product name, then into a folder for the comparison itself (still somewhat tbd). Given two sample build names, ar-23v1 and ar-24v1, the full folder structure might look something like this:
    edm-dataset-comparisons / ar-23v1 / ar-24v1 / qaqc_mismatch.csv
  • Bonus points: import the data into Postgres, into a table with columns for the dataset, build 1 name, build 2 name, and the comparison itself as JSON. This would make for painless integration with the QA app.

For inspiration, here's a Draft PR with a data model, and further explanations.

cc @damonmcc @fvankrieken @sf-dcp FYI, I don't have immediate plans to get started (though I will, if time permits) so don't hold back if you'd like to take this on.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing QA script and the Draft PR 524 for its data model and explanations. Then map the GitHub Action inputs to downloading two PLUTO builds, importing them into Postgres, generating comparison outputs, and uploading them to the edm-dataset-comparisons S3 bucket; done means the comparison workflow runs and cleans up its tables.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, github-actions, postgres, python
Domain
ci-cd, cloud, data, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.