NYCPlanning / NYCPlanning/data-engineering

Extract CSCL into "Working GDB"

Open
#2,110 2 comments 0 reactions 1 assignee View on GitHub

@damonmcc is already working on this.

Since Aug 10, 2026.

db-cscl
Dominant language
Python
Stars
43
Forks
3
Avg merge
23h 3m
Merged PRs (30d)
44

Description

Somewhat tangential from #2199 but related.

The current ETL tool gets aimed at a schema in the CSCL db, extracts into a file on the users computer (the "Working GDB"), and then runs the ETL pipeline from that gdb.

In terms of getting data out of the CSCL db and into... SOME other form, I'll leave that for #2199. We could potentially go straight from the CSCL db into postgres using ogr2ogr, or a foreign data wrapper, or something else. That means the GDB, from the perspective of our pipeline, is no longer needed as an input.

However, GR would still like this file created. To see how this happens in the C# application, poke around here. Since we're not starting the transformations from it though, this doesn't need to be run before we get data into the postgres db.

We could keep the same

  • Oracle -> Working GDB -> Postgres

Or instead go to Postgres first

  • Oracle -> Postgres -> Working GDB

Or parallel

  • Oracle -> Postgres -> dbt build
  • Oracle -> Working GDB

etc. My main concern is that interactions with the GDB take a while - around 20 min to load to postgres from GDB. So I'd rather not have that bottleneck the whole process.

Whether this happes in python, or with ogr2ogr, etc is not settled

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.