NYCPlanning / NYCPlanning/data-engineering
Ingest - archive datasets that haven't been run
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43
- Forks
- 3
- Avg merge
- 23h 3m
- Merged PRs (30d)
- 44
Description
33 datasets have an ingest template but have never been archived through ingest. Their latest/ is still entirely library output, so every consumer reads library data despite the template count saying they're migrated.
Running ingest for them is not a bulk operation. Archiving switches latest/ to ingest output, which silently changes which file a consumer resolves to, and the geometry column name often changes with it. That is what broke facilities, cpdb, green_fast_track and knownprojects in nightly QA after dcp_mappluto_wi was archived: nine references to b.wkb_geometry sat several lines above their LEFT JOIN dcp_mappluto_wi AS b, so a grep for the dataset name next to a geometry column found none of them.
7 are pinned by a consumer, so they fail loudly rather than silently. All in products/facilities/recipe.yml with file_type: csv: dcla_culturalinstitutions (see #2623), dcp_sfpsd, dohmh_daycare, fdny_firehouses, hhc_hospitals (see #1431), sbs_workforce1, uscourts_courts (blocked upstream, #1326).
26 have no pin, so archiving switches them to parquet with no warning. 17 of those are geospatial, and the templates disagree about the geometry column:
| template emits | datasets |
|---|---|
geom |
11 |
wkb_geometry |
dcp_zoningmapindex, dpr_park_access_zone, dsny_frequencies |
geometry |
dot_bridgehouses, dot_ferryterminals |
Several have 3 to 5 consuming SQL files. Each needs its consumers checked before archiving, and reading each join block rather than grepping.
9 are non-geospatial and low risk, with no geometry column to mismatch: council_members, dcp_pop_acs2010_demographic, dcp_pop_acs2010_economic, dcp_pop_acs2010_social, fbop_corrections, nycdoc_corrections, nysdoccs_corrections, nysed_activeinstitutions, sca_enrollment_capacity. The first four have no consuming SQL at all.
Suggested order: the 9 non-geospatial, then the 7 pinned ones (whose pins can come off once an archive exists), then the 17 geospatial one at a time.
admin/ops/clear_library_files_from_latest.py reports the current set as needs_ingest.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Run admin/ops/clear_library_files_from_latest.py to confirm the current needs_ingest set, then inspect products/facilities/recipe.yml and each listed dataset's consuming SQL join block. Archive the datasets in the suggested order, checking geometry columns and pinned consumers one at a time; done means all 33 datasets use ingest output without nightly QA breakage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sql
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100