NYCPlanning / NYCPlanning/data-engineering
fix COLP UID
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43
- Forks
- 3
- Avg merge
- 23h 3m
- Merged PRs (30d)
- 44
Description
what's wrong
geo_inputs.sql builds uid as MD5(CAST((dcas_ipis.*) AS text)) so it's a hash of the entire source row, including two columnds added during data loading (ogc_fid, data_library_version) that change every build
- Every UID changes every release. Across 10 published versions (2021→2026), zero UIDs persist, though ~16–17k records persist from one version to the next
- Duplicate rows
SELECT DISTINCTincreate_colp.sqlincludesuid, so identical source rows with differentogc_fidsurvive as duplicates (88 rows in 20260430, all originating in IPIS)
what we should probably do
- Base
uidon an explicit list of columns to create a natural key, not.*to make it version-stable and to letSELECT DISTINCTdrop the duplicates- should probably be
BBL+AGENCY+USECODE+LEASEDand generated duringcreate_colp.sql
- should probably be
- Move deduplication to ingest or early in the build (the duplicates are an IPIS source data)
- Resolve the few conflict groups that won't deduplicate: 3 differ on
owned/leased(OvsL, should probably remain distinct), 1 onu_f_use_code(already flattened downstream asfinalcom='D')
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with geo_inputs.sql and create_colp.sql, tracing how uid is built and how SELECT DISTINCT is applied. Check the IPIS-derived duplicate and conflict groups described in the issue, then define the explicit natural-key columns and deduplication point. Done means UIDs persist across releases and the identified duplicates are removed without incorrectly merging the owned/leased conflicts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100