NYCPlanning / NYCPlanning/data-engineering
FacDB - decide whether to take 5 years of dcla_culturalinstitutions changes
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43
- Forks
- 3
- Avg merge
- 23h 3m
- Merged PRs (30d)
- 44
Description
dcla_culturalinstitutions is pinned to file_type: csv in products/facilities/recipe.yml, reading a library archive from 2021. The comment says "migrated - Last updated in 2021", which is true of our archive but not of the source: the Socrata dataset (u35m-9t32) was last updated 2026-06-10.
Ingesting it is not a small addition. Comparing the current source against our archive:
| rows | 2,535 vs 2,308 (+227) |
| organizations in the source, not ours | 351 |
| organizations in ours, not in the source | 123 |
So 123 cultural institutions currently in FacDB would disappear and 351 would appear. Five years of closures, renames and re-registrations, not a clean append.
Columns are otherwise identical (17 each, only the geometry column name differs), so the mechanics are straightforward. The question is whether the churn is wanted in this version and whether downstream consumers should be warned about the 123 removals.
Whoever picks this up: facilities/facdb/sql/pipelines/dcla_culturalinstitutions.sql reads wkt::geometry, which is the csv's text column. An ingest archive supplies a real geometry instead, so that line needs updating alongside dropping the pin.
Found while auditing the file_type: csv pins in FacDB's recipe during the library-to-ingest migration (#1255). The other pinned datasets are genuinely frozen upstream; this is the one where the source moved on.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read products/facilities/recipe.yml and facilities/facdb/sql/pipelines/dcla_culturalinstitutions.sql, then compare the current Socrata source with the archived data and identify downstream consumers. Done means the churn decision is documented; if the source is updated, the csv pin is removed, the geometry handling is adjusted for the ingest archive, and downstream impact is addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql, yaml
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100