NYCPlanning / NYCPlanning/data-engineering
versioning of dcp addresspoints for use in gru qaqc
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43
- Forks
- 3
- Avg merge
- 23h 3m
- Merged PRs (30d)
- 44
Description
This maybe belongs in the db-gru-qaqc repo, but putting here for visiblity.
For all datasets, the gru code assumes that they've been archived in recipes with "YY{a-d}" scheme. However, addresspoints doesn't actually follow this scheme (at least in edm-publishing/datasets), it's simply by date.
However, when we ran library in the past, we'd just pull it from the staging folder and manually specify that the version was "24d". Now, new ingest code actually finds the "latest"/max versioned foldername in datasets folder and bases version on that. A little more certainty in terms of what data we're referring to. But now tougher to link dcpaddresspoints to quarterly releases for the sake of gru qaqc.
I think ideally, we add some special import logic in the gru repo (or just in recipes connector), that
- finds the date of archival of one of the "actual" 24d release datasets (or just uses hardcoded dates - Nov 1 or something for "d") to figure out how fresh data should be
- uses that date to try to look up a corresponding version of dcp_addresspoints to use, and errors if there's not one within... 1 month? 2 months?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the new ingest code and the recipes connector, then compare how release dataset versions and dcp_addresspoints archival folders are selected. Clarify whether the logic belongs in db-gru-qaqc or this repository, define the date or version matching window and failure behavior, and confirm the result against quarterly releases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100