NYCPlanning / NYCPlanning/data-engineering
GDE data flow improvements
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43
- Forks
- 3
- Avg merge
- 23h 3m
- Merged PRs (30d)
- 44
Description
Project Description
Since GDE is upstream of other teams in ITD and maintains some of the relevant automations, we will improve and standardize how data flows to those teams for QA, distribution, and usage.
TImeline
Week of 2/16
- DE meets with AE to share how we use github actions for data flow
- https://github.com/NYCPlanning/data-engineering/issues/649
Future
- #641
- #825
- #227
- see sub-issues below
Ideas from 2/1 meeting
- GDE <> SPM
- avoid duplicate issue creation in
edm-data-operations
- avoid duplicate issue creation in
- DE <> GIS
- when ready for QA, promote a build and create a GH issue for GIS
- use a staging directory for data to QA
- DE <> AE
- how to update data in Zoning API backend DB
- how to communicate/validate AE’s data expectations
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No implementation files, tests, or single entry point are identified. Start by reviewing the completed issues #641, #825, #227, and linked issue #649, then clarify which data flow improvement remains and which teams and automation it affects. Done should be defined as an agreed, standardized flow for the selected QA, distribution, or usage process.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions
- Domain
- data-engineering, devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100