NYCPlanning / NYCPlanning/data-engineering
Extract CSCL into "Working GDB"
@damonmcc is already working on this.
Since Aug 10, 2026.
- Dominant language
- Python
- Stars
- 43
- Forks
- 3
- Avg merge
- 23h 3m
- Merged PRs (30d)
- 44
Description
Somewhat tangential from #2199 but related.
The current ETL tool gets aimed at a schema in the CSCL db, extracts into a file on the users computer (the "Working GDB"), and then runs the ETL pipeline from that gdb.
In terms of getting data out of the CSCL db and into... SOME other form, I'll leave that for #2199. We could potentially go straight from the CSCL db into postgres using ogr2ogr, or a foreign data wrapper, or something else. That means the GDB, from the perspective of our pipeline, is no longer needed as an input.
However, GR would still like this file created. To see how this happens in the C# application, poke around here. Since we're not starting the transformations from it though, this doesn't need to be run before we get data into the postgres db.
We could keep the same
- Oracle -> Working GDB -> Postgres
Or instead go to Postgres first
- Oracle -> Postgres -> Working GDB
Or parallel
- Oracle -> Postgres -> dbt build
- Oracle -> Working GDB
etc. My main concern is that interactions with the GDB take a while - around 20 min to load to postgres from GDB. So I'd rather not have that bottleneck the whole process.
Whether this happes in python, or with ogr2ogr, etc is not settled
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.