NYCPlanning / NYCPlanning/data-engineering
CSCL - ETL UI/Ergonomics
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43
- Forks
- 3
- Avg merge
- 23h 3m
- Merged PRs (30d)
- 44
Description
Bit of a stub to be brainstormed - how does GR actually interact with this?
- web dashboard?
- how do they "load" data?
- How do they choose which outputs to create? (or maybe not an issue if build times are fast enough)
- how do they receive error reports?
Very rough user story.
- ETL is used during "production activities" in GR. Production activities last a week or two, and are the process of preparing a geosupport release (i.e. "26A")
- Starting "production activities", a user will create a new schema (?) in the oracle cscl database, through the CSCL maintenance tool in ArcMap/ArcPro
- User specifies the ETL tool to target a specific schema (ideally, from a searchable dropdown?)
- ETL begins. A lot of requirements here, but will keep that in other issues as this issue is more about the UI itself
- User downloads outputs, error reports
- User does some form of QA (for now, outside of universe of this ETL)
- User makes edits to either the same schema in CSCL DB or a new one
- User triggers new ETL from chosen schema
- Repeat above cycle until outputs are approved
- User "merges" schema in CSCL db into main/production/etc
- User runs grid generation, etc (other downstream processes begin)
Assorted requirements for above
- if user has a dropdown for schemas to extract from, the UI itself, wherever it is hosted, needs access to the cscl database. This would need a static IP address for wherever this is hosted, on citynet, and credentials that need to be cycled every 90 days. see (TODO MAKE SUBISSUE)
- (TODO - subissue for file downloads)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
This issue names no implementation files, tests, or entry points; begin by clarifying where the ETL UI is hosted and how users access schemas, outputs, and error reports. Define the production workflow and its UI requirements, including schema selection, repeated runs, downloads, and completion criteria, before locating implementation work or splitting follow-up issues.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering, database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100