NYCPlanning / NYCPlanning/data-engineering
edm-recipes catalog
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43
- Forks
- 3
- Avg merge
- 23h 3m
- Merged PRs (30d)
- 44
Description
Would be nice to get away from "latest" as a valid version and instead use a catalog (say, a parquet file in datasets folder or something), as well as implement a similar thing for raw datasets (so that we don't have to scrape the timestamps). That would also get us away from updating configs with check timestamps
Problem with this is that we can't have a "real-time" catalog (that supports concurrent writes) without some persistent server or db -> either a postgres db that gets logged to as the "official" catalog or use iceberg. So this needs quite some scoping
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the datasets folder and the configuration files that currently use “latest” and check timestamps. Scope how catalogs for EDM recipes and raw datasets should work, including concurrent writes and the PostgreSQL or Iceberg alternatives mentioned. Done requires an agreed design and implementation plan before coding begins.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100