NYCPlanning / NYCPlanning/data-engineering

FacDB - decide whether to take 5 years of dcla_culturalinstitutions changes

Open
#2,623 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

db-facilities
Dominant language
Python
Stars
43
Forks
3
Avg merge
23h 3m
Merged PRs (30d)
44

Description

dcla_culturalinstitutions is pinned to file_type: csv in products/facilities/recipe.yml, reading a library archive from 2021. The comment says "migrated - Last updated in 2021", which is true of our archive but not of the source: the Socrata dataset (u35m-9t32) was last updated 2026-06-10.

Ingesting it is not a small addition. Comparing the current source against our archive:

rows 2,535 vs 2,308 (+227)
organizations in the source, not ours 351
organizations in ours, not in the source 123

So 123 cultural institutions currently in FacDB would disappear and 351 would appear. Five years of closures, renames and re-registrations, not a clean append.

Columns are otherwise identical (17 each, only the geometry column name differs), so the mechanics are straightforward. The question is whether the churn is wanted in this version and whether downstream consumers should be warned about the 123 removals.

Whoever picks this up: facilities/facdb/sql/pipelines/dcla_culturalinstitutions.sql reads wkt::geometry, which is the csv's text column. An ingest archive supplies a real geometry instead, so that line needs updating alongside dropping the pin.

Found while auditing the file_type: csv pins in FacDB's recipe during the library-to-ingest migration (#1255). The other pinned datasets are genuinely frozen upstream; this is the one where the source moved on.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read products/facilities/recipe.yml and facilities/facdb/sql/pipelines/dcla_culturalinstitutions.sql, then compare the current Socrata source with the archived data and identify downstream consumers. Done means the churn decision is documented; if the source is updated, the csv pin is removed, the geometry handling is adjusted for the ingest archive, and downstream impact is addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql, yaml
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.