scrapinghub / scrapinghub/python-scrapinghub
Method to iterate all jobs/items/whatever
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 205
- Forks
- 62
- PR merge metrics
- No merged PRs in 30d
Description
The AWS boto client has a Paginator to help iterating within api result limits, which while clunky is very nice to have since it's hard to get it wrong.
A method to iterate/list all results or else a Paginator that hides the pagination parameters which are easy to get wrong would be super helpful. The 1000 job limit is an issue in all projects I've needed to use this in.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the client entry points for listing jobs and other API results, then compare their pagination behavior with the AWS boto Paginator described in the issue. Done means callers can iterate beyond the 1,000-job limit without manually handling error-prone pagination parameters.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100