NYCPlanning / NYCPlanning/data-engineering

Ingest - handling "custom" scripts

Open
#1,533 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
43
Forks
3
Avg merge
23h 3m
Merged PRs (30d)
44

Description

Follow up to #833 to limit scope a little bit

We'll need to figure out how we want to handle these. I don't know that there will be a one-size-fits-all solution.

Specific script sources
  • hra_centers (scrape/soup -> probably custom script source for now) this is defunct
  • moeo_socialservicesitelocations (joins 4 datasets. could/should be split into 5 recipe datasets) has its own issue #1265
    • would be four simple recipes, and then one novel recipe to combine them. OR we just make four recipes, and save the unioning for facdb
  • nycdoc_corrections (beautiful soup source. but just one webpage, nothing dynamic)
    • maybe we could get by with read_html? This one would be good to explore first I think
  • nycoc_checkbook (special script. Connector!)
    • likely custom connector
  • usfws_nyc_wetlands (eventually could break into multiple that get merged, for now just keep as custom script source)
    • break into multiple, union in gft
  • uscourts_courts - funky one. Hits two endpoints, appends, deduplicates. Not could break into two, but not sure
    • break into multiple, union in pipelines

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the follow-up context in #833 and the listed sources, especially nycdoc_corrections, nycoc_checkbook, usfws_nyc_wetlands, and uscourts_courts. Determine which sources can become standard recipes, connectors, or split-and-union pipelines; done means the handling and scope for each remaining source are agreed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.