NYCPlanning / NYCPlanning/data-engineering

CSCL - ETL UI/Ergonomics

Open
#1,841 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

db-cscl
Dominant language
Python
Stars
43
Forks
3
Avg merge
23h 3m
Merged PRs (30d)
44

Description

Bit of a stub to be brainstormed - how does GR actually interact with this?

  • web dashboard?
  • how do they "load" data?
  • How do they choose which outputs to create? (or maybe not an issue if build times are fast enough)
  • how do they receive error reports?

Very rough user story.

  • ETL is used during "production activities" in GR. Production activities last a week or two, and are the process of preparing a geosupport release (i.e. "26A")
  • Starting "production activities", a user will create a new schema (?) in the oracle cscl database, through the CSCL maintenance tool in ArcMap/ArcPro
  • User specifies the ETL tool to target a specific schema (ideally, from a searchable dropdown?)
  • ETL begins. A lot of requirements here, but will keep that in other issues as this issue is more about the UI itself
  • User downloads outputs, error reports
  • User does some form of QA (for now, outside of universe of this ETL)
  • User makes edits to either the same schema in CSCL DB or a new one
  • User triggers new ETL from chosen schema
  • Repeat above cycle until outputs are approved
  • User "merges" schema in CSCL db into main/production/etc
  • User runs grid generation, etc (other downstream processes begin)

Assorted requirements for above

  • if user has a dropdown for schemas to extract from, the UI itself, wherever it is hosted, needs access to the cscl database. This would need a static IP address for wherever this is hosted, on citynet, and credentials that need to be cycled every 90 days. see (TODO MAKE SUBISSUE)
  • (TODO - subissue for file downloads)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

This issue names no implementation files, tests, or entry points; begin by clarifying where the ETL UI is hosted and how users access schemas, outputs, and error reports. Define the production workflow and its UI requirements, including schema selection, repeated runs, downloads, and completion criteria, before locating implementation work or splitting follow-up issues.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.