NYCPlanning / NYCPlanning/data-engineering

ZTL - make sure comparison is to previous version

Open
#1,727 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

db-zoningtaxlots QA
Dominant language
Python
Stars
43
Forks
3
Avg merge
23h 3m
Merged PRs (30d)
44

Description

There's a couple ways to go about this, but we've had a few times now were GIS builds ZTL, but the previous version was never archived with ingest (or even published from draft to publish often, as was the case this time). This often doesn't come up until they've already done some QA, so this has been a big waste of time for them.

There are a few things that would work/help. Worth brainstorming a little as to what feels most appropriate (and maybe check with GIS as well)

  1. a data check - check that the "previous" version (latest from edm-recipes) is only one month less than the current version
  2. pull from edm-publishing latest instead of ingest (gets rid of a step, though in the case
  3. make the versions much more apparent in the qa files (filenames?)
  4. make sure that GIS only publishes ZTL from publish folder rather than draft as it seems that they do now (since last month was never published
  5. add something of job to the ztl dataloading action that
    • makes sure previous draft was actually published
    • archives ztl from publish-latest into edm-recipes

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the ZTL dataloading action and how it uses edm-recipes, edm-publishing, and the draft and publish folders. Compare the proposed data check, source change, QA filenames, and archive steps with the current workflow; done means an agreed approach prevents QA from proceeding against an unavailable or incorrect previous version.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.