NYCPlanning / NYCPlanning/data-engineering

KPDB - phasing uses current_date so a pinned recipe doesn't rebuild the same numbers

Open Beginner friendly
#2,629 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

db-knownprojects
Dominant language
Python
Stars
43
Forks
3
Avg merge
23h 3m
Merged PRs (30d)
44

Description

Found while comparing the new CPP housing growth outputs against an earlier PoC pipeline (#2578).

sql/combine.sql:122-139 phases EDC Projected Projects by comparing each project's build_year against the wall clock:

(CASE
    WHEN build_year::numeric <= date_part('year', current_date) + 5
        THEN 1
    ELSE 0
END) AS prop_within_5_years,

All three buckets do this. So the boundaries move every January no matter what recipe.yml pins, and rebuilding the exact same recipe in a later calendar year gives different numbers. That hits every KPDB geography aggregation, not just the CPP outputs.

It's small but not nothing: 3,044 of 404,939 projected units on the 26Q2 build, about 0.8%. The reason to fix it isn't the size, it's that a pinned recipe should rebuild to the same answer.

Run today (2026-09-14) within_5_years here means build_year <= 2031 and from_5_to_10_years means 2032 through 2036, so a CPP column labeled 2026-2035 picks up EDC sites through 2036.

The fix is to anchor the comparison to a build year that comes from the recipe instead of the clock. This is the only place current_date reaches KPDB phasing. Everything else is status-based or uses shares supplied by the source file, which is a separate and known limitation (README, "These buckets are relative, not calendar dates").

review/review_dob.sql:108 also uses current_date, but that's a review table rather than product output, so it can be left alone or fixed in the same pass.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in sql/combine.sql:122-139 and trace how the build year is pinned in recipe.yml. Check the three phasing buckets and confirm that rebuilding the same pinned recipe produces the same boundaries and values; review/review_dob.sql:108 is optional because it affects a review table rather than product output.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.