NYCPlanning / NYCPlanning/data-engineering

DevDB - Corrections QOL changes

Open
#1,698 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

db-developments
Dominant language
Python
Stars
43
Forks
3
Avg merge
23h 3m
Merged PRs (30d)
44

Description

Coming up in convos related to #1601

We plan on sitting down with Housing to revamp QA just a little more in depth after #1605 is built, but for now there are some low hanging fruit. SL shared HDB_Setup.docx which documents some "preprocessing" they do before QA - we should just do this for them and give them what they need. This includes

  • providing a QA file that is already filtered to post2010 (created/updated) and doesn't include inactives. They currently start their process with "housing.csv" which includes all records
  • they then join FINAL_qaqc to this table. This join can be done upstream (and should probably replace this qaqc file)

There are a bunch of other details in there, but we agreed that it'd make sense to keep the slightly more complex logic till we have a chance to go through QA together, mainly with bandwidth of both teams for writing requirements and implementation, with the focus in July being on #1601

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading HDB_Setup.docx and the context in #1601 and #1605. Trace where housing.csv and FINAL_qaqc are produced, then confirm the intended post-2010 and inactive-record filters with the team. Done means a QA-ready output replaces the current preprocessing and upstream join, with the remaining logic deferred until requirements are agreed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.