NYCPlanning / NYCPlanning/data-engineering
Ingest - handling "custom" scripts
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43
- Forks
- 3
- Avg merge
- 23h 3m
- Merged PRs (30d)
- 44
Description
Follow up to #833 to limit scope a little bit
We'll need to figure out how we want to handle these. I don't know that there will be a one-size-fits-all solution.
Specific script sources
-
hra_centers (scrape/soup -> probably custom script source for now)this is defunct -
moeo_socialservicesitelocations (joins 4 datasets. could/should be split into 5 recipe datasets)has its own issue #1265would be four simple recipes, and then one novel recipe to combine them. OR we just make four recipes, and save the unioning for facdb
- nycdoc_corrections (beautiful soup source. but just one webpage, nothing dynamic)
- maybe we could get by with read_html? This one would be good to explore first I think
- nycoc_checkbook (special script. Connector!)
- likely custom connector
- usfws_nyc_wetlands (eventually could break into multiple that get merged, for now just keep as custom script source)
- break into multiple, union in gft
- uscourts_courts - funky one. Hits two endpoints, appends, deduplicates. Not could break into two, but not sure
- break into multiple, union in pipelines
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the follow-up context in #833 and the listed sources, especially nycdoc_corrections, nycoc_checkbook, usfws_nyc_wetlands, and uscourts_courts. Determine which sources can become standard recipes, connectors, or split-and-union pipelines; done means the handling and scope for each remaining source are agreed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100