scrapinghub / scrapinghub/python-scrapinghub

Add async support

Open
#169 8 comments 2 reactions 4 assignees View on GitHub

@VMRuiz is already working on this.

Since Jan 17, 2023.

Dominant language
Python
Stars
205
Forks
62
PR merge metrics
No merged PRs in 30d

Description

Scrapy now supports calling async methods from many components. It would be great to provide the option to call Scrapinghub API using async methods to avoid either blocking calls or having to work with twisted deferreds and threads.

In order to do that, there are several design decisions that needs to be taken:

Define Async interface
  • Do we add an async version for each method already available?
  • Do we create an AsyncScrapinghubClient that only provides support for async methods?
How to implement async methods
Possible obstacles?

If we want to use python-scrapinghub in standalone mode, will it create their own task loop? or should it expect it to be already created?
If we run it inside Scrapy, could this cause conflict issues?

I don't have much experience designing async libs so It would be great if someone with a better understanding could provide some pointers here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.