CSCL ETL

CSCL Modernization Azure DevOps Project
CSCL Project Plan Archive SharePoint folder
CSCL ETL SharePoint folder

CSCL Issue Tracking excel file
Legacy ETL Design doc

What the pipeline needs to do

Very high level, this consists of

• Extraction of data from an Oracle (geo)DB hosted at OTI (both into a file gdb and our build engine)
• Creation of file outputs. LION is the primary output - one row per linear feature in the city, but there are many
  • 32 files that go into geosupport (really 20 files but 3 of them are split out by borough). These almost all densely formatted text files
  • A few geodatabases - one for LION, one for all district boundaries
  • log file with warnings/errors from the etl process
• A user interface for the above
• cloud infrastructure (and DE code infrastructure) to facilitate the above

Implementation

Transformations and file outputs

Primary focus is first on the geosupport files - dense text files. These all have a similar workflow

• Referencing the existing ETL docs, implement transformations to generate the outputs
• Compare these to production outputs and isolate differences
• Iterate with GR to discuss and resolve these differences. They may stem from source data issues, bugs in the production code, or bugs in our pipeline
• Run integration tests with GSS once the differences are "acceptable". Won't get into that here, but basically once all remaining differences have been characterized and either resolved or deemed as not a problem

For the above, we've planned this in three buckets

• #1820
• #1843
• #2195

After these are done, we have the non-geosupport outputs #2196

Architecture

[stub]

UI

[stub]