NYCPlanning / NYCPlanning/data-engineering

fix COLP UID

Open
#2,510 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

db-colp
Dominant language
Python
Stars
43
Forks
3
Avg merge
23h 3m
Merged PRs (30d)
44

Description

what's wrong

geo_inputs.sql builds uid as MD5(CAST((dcas_ipis.*) AS text)) so it's a hash of the entire source row, including two columnds added during data loading (ogc_fid, data_library_version) that change every build

  1. Every UID changes every release. Across 10 published versions (2021→2026), zero UIDs persist, though ~16–17k records persist from one version to the next
  2. Duplicate rows SELECT DISTINCT in create_colp.sql includes uid, so identical source rows with different ogc_fid survive as duplicates (88 rows in 20260430, all originating in IPIS)

what we should probably do

  • Base uid on an explicit list of columns to create a natural key, not .* to make it version-stable and to let SELECT DISTINCT drop the duplicates
    • should probably be BBL+AGENCY+USECODE+LEASED and generated during create_colp.sql
  • Move deduplication to ingest or early in the build (the duplicates are an IPIS source data)
  • Resolve the few conflict groups that won't deduplicate: 3 differ on owned/leased (O vs L, should probably remain distinct), 1 on u_f_use_code (already flattened downstream as finalcom='D')

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with geo_inputs.sql and create_colp.sql, tracing how uid is built and how SELECT DISTINCT is applied. Check the IPIS-derived duplicate and conflict groups described in the issue, then define the explicit natural-key columns and deduplication point. Done means UIDs persist across releases and the identified duplicates are removed without incorrectly merging the owned/leased conflicts.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.