scrapinghub / scrapinghub/python-scrapinghub
Long intervals during resource iteration can lead to issues
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 205
- Forks
- 62
- PR merge metrics
- No merged PRs in 30d
Description
Hello.
Recently there was this issue https://github.com/scrapinghub/python-scrapinghub/issues/121 for which a batch read workaround was implemented. I am now experiencing from what I believe to be same or similar issue but now while using JSON instead of msgpack. Basically when I do for item in job.items.iter(..., count=X, ...): if there are long intervals during iteration the count can end up being ignored. I was able to reproduce it with the following snippet:
sh_client = ScrapinghubClient(APIKEY, use_msgpack=False)
take = 10_000
job_id = '168012/276/1'
for i, item in enumerate(sh_client.get_job(job_id).items.iter(count=take, meta='_key')):
print(f'\r{i} ({item["_key"]})', end='')
if i == 3000:
print('\nsleeping')
time.sleep(60*3)
if i > take:
print('\nWTF')
break
With the sleep part removed the WTF section does not fire and the iterator stops on 168012/276/1/9999th item.
This seem to be more of a ScrapyCloud API platform problem but I am reporting it here to track nonetheless.
For now I am assuming resource/collections iteration is not robust if any delays are possible client side during retrieval (I haven't tested any other potential issues) and I will try either preloading all at once (.list()) or using .list_iter() when makes sense as a habit.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the resource collection entry point used by job.items.iter(count=take, meta='_key'), and reproduce the JSON case with use_msgpack=False and the three-minute delay. Trace how iteration requests preserve count across delayed retrieval; done means the iterator stops at the requested limit instead of exceeding it, with the behavior covered by the repository's existing tests if applicable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100