NYCPlanning / NYCPlanning/data-engineering
KPDB - phasing uses current_date so a pinned recipe doesn't rebuild the same numbers
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43
- Forks
- 3
- Avg merge
- 23h 3m
- Merged PRs (30d)
- 44
Description
Found while comparing the new CPP housing growth outputs against an earlier PoC pipeline (#2578).
sql/combine.sql:122-139 phases EDC Projected Projects by comparing each project's build_year against the wall clock:
(CASE
WHEN build_year::numeric <= date_part('year', current_date) + 5
THEN 1
ELSE 0
END) AS prop_within_5_years,
All three buckets do this. So the boundaries move every January no matter what recipe.yml pins, and rebuilding the exact same recipe in a later calendar year gives different numbers. That hits every KPDB geography aggregation, not just the CPP outputs.
It's small but not nothing: 3,044 of 404,939 projected units on the 26Q2 build, about 0.8%. The reason to fix it isn't the size, it's that a pinned recipe should rebuild to the same answer.
Run today (2026-09-14) within_5_years here means build_year <= 2031 and from_5_to_10_years means 2032 through 2036, so a CPP column labeled 2026-2035 picks up EDC sites through 2036.
The fix is to anchor the comparison to a build year that comes from the recipe instead of the clock. This is the only place current_date reaches KPDB phasing. Everything else is status-based or uses shares supplied by the source file, which is a separate and known limitation (README, "These buckets are relative, not calendar dates").
review/review_dob.sql:108 also uses current_date, but that's a review table rather than product output, so it can be left alone or fixed in the same pass.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in sql/combine.sql:122-139 and trace how the build year is pinned in recipe.yml. Check the three phasing buckets and confirm that rebuilding the same pinned recipe produces the same boundaries and values; review/review_dob.sql:108 is optional because it affects a review table rather than product output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 74/100