Custom scrapers as harvesters
- Dominant language
- No language data
- Stars
- 39
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Currently harvesters work based on the format at the other end of the wire. So, there are harvesters for datajson files, and harvesters that use the CKAN API.
It might be interesting, and useful, to allow for a harvester type that runs scrapers. The scrapers could extend a specific object, or have a particular interface, and could then be triggered as a normal harvester and extract its data from anywhere.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are named. Start by reviewing the existing datajson and CKAN API harvester implementations, then determine the required scraper interface and how it would be triggered as a normal harvester. Done would require an agreed design and a working custom-scraper harvester path.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100