NYCPlanning / NYCPlanning/data-engineering
DevDB data check - ensure all new corrections applied
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43
- Forks
- 3
- Avg merge
- 23h 3m
- Merged PRs (30d)
- 44
Description
Manual corrections have specified their date of creation. DevDB also has an env var called "capture date", the effective date of build (date up to which jobs are included in the build). (there's also an issue to create that env var programmatically rather than set it manually, see #696 )
In a build cycle, we originally build with no added corrections, then housing edits the manual_corrections.csv file and we rebuild. If these corrections are not written correctly and are not applied, this should throw an error so that we don't send data back to housing (or publish it) without the new corrections applied.
So essentially, we could just have a data check that checks if any corrections not applied have edited_date > capture_date, and fail the build (or loudly flag, etc) if so.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the build's handling of manual_corrections.csv and the DevDB capture-date environment variable, then read the existing data checks and the context in issue #696. Done means the build fails or loudly flags an unapplied correction whose edited_date is later than the capture date, before data is sent or published.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100