NYCPlanning / NYCPlanning/data-engineering

Ingest - archive datasets that haven't been run

Open
#2,624 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
43
Forks
3
Avg merge
23h 3m
Merged PRs (30d)
44

Description

33 datasets have an ingest template but have never been archived through ingest. Their latest/ is still entirely library output, so every consumer reads library data despite the template count saying they're migrated.

Running ingest for them is not a bulk operation. Archiving switches latest/ to ingest output, which silently changes which file a consumer resolves to, and the geometry column name often changes with it. That is what broke facilities, cpdb, green_fast_track and knownprojects in nightly QA after dcp_mappluto_wi was archived: nine references to b.wkb_geometry sat several lines above their LEFT JOIN dcp_mappluto_wi AS b, so a grep for the dataset name next to a geometry column found none of them.

7 are pinned by a consumer, so they fail loudly rather than silently. All in products/facilities/recipe.yml with file_type: csv: dcla_culturalinstitutions (see #2623), dcp_sfpsd, dohmh_daycare, fdny_firehouses, hhc_hospitals (see #1431), sbs_workforce1, uscourts_courts (blocked upstream, #1326).

26 have no pin, so archiving switches them to parquet with no warning. 17 of those are geospatial, and the templates disagree about the geometry column:

template emits datasets
geom 11
wkb_geometry dcp_zoningmapindex, dpr_park_access_zone, dsny_frequencies
geometry dot_bridgehouses, dot_ferryterminals

Several have 3 to 5 consuming SQL files. Each needs its consumers checked before archiving, and reading each join block rather than grepping.

9 are non-geospatial and low risk, with no geometry column to mismatch: council_members, dcp_pop_acs2010_demographic, dcp_pop_acs2010_economic, dcp_pop_acs2010_social, fbop_corrections, nycdoc_corrections, nysdoccs_corrections, nysed_activeinstitutions, sca_enrollment_capacity. The first four have no consuming SQL at all.

Suggested order: the 9 non-geospatial, then the 7 pinned ones (whose pins can come off once an archive exists), then the 17 geospatial one at a time.

admin/ops/clear_library_files_from_latest.py reports the current set as needs_ingest.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run admin/ops/clear_library_files_from_latest.py to confirm the current needs_ingest set, then inspect products/facilities/recipe.yml and each listed dataset's consuming SQL join block. Archive the datasets in the suggested order, checking geometry columns and pinned consumers one at a time; done means all 33 datasets use ingest output without nightly QA breakage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.