NYCPlanning / NYCPlanning/data-engineering

DevDB data check - ensure all new corrections applied

Open
#1,209 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

db-developments QA
Dominant language
Python
Stars
43
Forks
3
Avg merge
23h 3m
Merged PRs (30d)
44

Description

Manual corrections have specified their date of creation. DevDB also has an env var called "capture date", the effective date of build (date up to which jobs are included in the build). (there's also an issue to create that env var programmatically rather than set it manually, see #696 )

In a build cycle, we originally build with no added corrections, then housing edits the manual_corrections.csv file and we rebuild. If these corrections are not written correctly and are not applied, this should throw an error so that we don't send data back to housing (or publish it) without the new corrections applied.

So essentially, we could just have a data check that checks if any corrections not applied have edited_date > capture_date, and fail the build (or loudly flag, etc) if so.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Locate the build's handling of manual_corrections.csv and the DevDB capture-date environment variable, then read the existing data checks and the context in issue #696. Done means the build fails or loudly flags an unapplied correction whose edited_date is later than the capture date, before data is sent or published.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.