Owner

scrapy

10 indexed repositories · View on GitHub

scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.

Python · 64407 stars
scrapyd
A service daemon to run Scrapy spiders

Python · 3098 stars
parsel
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

Python · 1354 stars
w3lib
Python library of web-related functions

HTML · 417 stars
cssselect
CSS Selectors for Python

Python · 310 stars
protego
A pure-Python robots.txt parser with support for modern conventions.

DIGITAL Command Language · 94 stars
itemadapter
Common interface for data container classes

Python · 69 stars
itemloaders
Library to populate items using XPath and CSS with a convenient API

Python · 49 stars
scrapy-lint
A linter for Scrapy projects.

Python · 22 stars
form2request
Python library to build HTTP requests out of HTML forms

Python · 5 stars

Beginner friendly Everything

Open All

`data_path()` fails under Scrapyd because there is no `scrapy.cfg` Open

Difficulty 3/5 1-2 days Newbie friendliness 55/100

scrapy/scrapyd#575 · Opened Sep 8, 2026
Removing the non-asyncio reactor support Open

asyncio

scrapy/scrapy#8111 · 2 comments · 1 reaction · 1 assignee · Opened Sep 3, 2026
Support crash-tolerant crawls Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

scrapy/scrapy#7978 · 2 comments · 1 reaction · Opened Aug 11, 2026
docs/options.rst does not match the code Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 76/100

scrapy/scrapy-lint#178 · Opened Aug 9, 2026
Support non-frozen requirements Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 48/100

scrapy/scrapy-lint#175 · Opened Aug 9, 2026
scrapy-zyte-api rule: automap params on a request whose response is unused Open

Difficulty 4/5 3-5 days Newbie friendliness 52/100

scrapy/scrapy-lint#169 · Opened Aug 9, 2026
scrapy-zyte-api rule: automap params set without provider params Open

Difficulty 2/5 1-3 hours Newbie friendliness 74/100

scrapy/scrapy-lint#168 · Opened Aug 9, 2026
New rule: redundant add-on setting entry Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

scrapy/scrapy-lint#166 · Opened Aug 9, 2026
Add-on-aware effective defaults Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

scrapy/scrapy-lint#165 · Opened Aug 9, 2026
New rule: wrong add-on order Open

Difficulty 3/5 1-2 days Newbie friendliness 72/100

scrapy/scrapy-lint#164 · Opened Aug 9, 2026
New rule: invalid component deactivation Open

Difficulty 3/5 1-2 days Newbie friendliness 65/100

scrapy/scrapy-lint#163 · Opened Aug 9, 2026
Report unused ignore comments Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 65/100

scrapy/scrapy-lint#147 · Opened Aug 8, 2026
Add --add-ignore Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 68/100

scrapy/scrapy-lint#146 · Opened Aug 8, 2026
Add clone to item interface Open

Difficulty 3/5 1-2 days Newbie friendliness 38/100

scrapy/itemadapter#124 · 4 comments · Opened Aug 7, 2026
Could I get the lastmod of an url in function parse when using SitemapSpider? Open

Difficulty 3/5 1-2 days Newbie friendliness 48/100

scrapy/scrapy#7852 · 3 comments · Opened Aug 2, 2026
Rebase `Headers` to `multidict.CIMultiDict` Open

discuss enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

scrapy/scrapy#7850 · Opened Aug 1, 2026
Deprecate or drop PyPy support Open

discuss

Difficulty 5/5 Over a week Newbie friendliness 35/100

scrapy/scrapy#7849 · 3 comments · Opened Aug 1, 2026
Add S3 append support Open

enhancement S3

Difficulty 5/5 Over a week Newbie friendliness 35/100

scrapy/scrapy#7822 · 5 comments · Opened Jul 30, 2026
is_asyncio_available() is not fully supported in extension __init__() Open

asyncio bug

scrapy/scrapy#7812 · 1 assignee · Opened Jul 29, 2026
Update setuptools version Open

Difficulty 2/5 1-3 hours Newbie friendliness 74/100

scrapy/scrapyd#573 · 3 comments · Opened Jul 29, 2026
Add a way to keep the engine running after the spider has finished Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

scrapy/scrapy#7791 · 2 comments · Opened Jul 27, 2026
Let downloader middlewares process exceptions from process_response() Open

discuss enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

scrapy/scrapy#7790 · Opened Jul 27, 2026
remove_tags() leaks markup when an attribute value contains '>' (and the obvious fix is a ReDoS) Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

scrapy/w3lib#284 · 1 comment · Opened Jul 22, 2026
Add comprehensive developer docs Open

documentation enhancement

Difficulty 4/5 3-5 days Newbie friendliness 48/100

scrapy/cssselect#171 · Opened Jul 21, 2026
Embedded `ipykernel` Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 30/100

scrapy/scrapy#7752 · 2 comments · Opened Jul 20, 2026
Cleanup module-level docstrings Open

cleanup docs

Difficulty 4/5 3-5 days Newbie friendliness 45/100

scrapy/scrapy#7699 · 1 reaction · Opened Jun 30, 2026
New rule: urlparse → urlparse_cached Open

new rule

Difficulty 2/5 1-3 hours Newbie friendliness 68/100

scrapy/scrapy-lint#117 · Opened Jun 26, 2026
Revive/update `--profile` command line option for more convenient use Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 45/100

scrapy/scrapy#7636 · 2 comments · 2 reactions · Opened Jun 18, 2026
Request for user feedback: the state of S3 support Open

discuss S3

Difficulty 5/5 Over a week Newbie friendliness 35/100

scrapy/scrapy#7607 · Opened Jun 12, 2026
Add asynchronous support to `spider_idle` signal and other synchronous signals Open

asyncio enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

scrapy/scrapy#7568 · 3 comments · Opened Jun 3, 2026
Bug: form2request ignores submit buttons without attribute Open

Difficulty 1/5 1-3 hours Newbie friendliness 85/100

scrapy/form2request#12 · Opened May 31, 2026
HTTPS proxy certificate checking Open

discuss https

Difficulty 4/5 3-5 days Newbie friendliness 45/100

scrapy/scrapy#7500 · 5 comments · Opened May 8, 2026
Plan next steps for `ensure_awaitable()` Open

cleanup

Difficulty 5/5 Over a week Newbie friendliness 25/100

scrapy/scrapy#7407 · 1 comment · Opened Apr 6, 2026
Investigate running a shell without a separate thread with an async REPL Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

scrapy/scrapy#7272 · Opened Feb 20, 2026
Add an FTP download handler that doesn't use Twisted Open

asyncio enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

scrapy/scrapy#7261 · 1 reaction · Opened Feb 15, 2026
Investigate non-Twisted-based alternatives to `TelnetConsole` Open

asyncio enhancement

Difficulty 5/5 Over a week Newbie friendliness 30/100

scrapy/scrapy#7251 · 2 comments · Opened Feb 12, 2026
Get rid of get_event_loop() Open

asyncio enhancement

Difficulty 4/5 3-5 days Newbie friendliness 48/100

scrapy/scrapy#7201 · 4 comments · Opened Dec 26, 2025
Refactor/clean up `handle_httpstatus_list`/`HTTPERROR_ALLOWED_CODES` etc. Open

cleanup discuss

Difficulty 5/5 Over a week Newbie friendliness 25/100

scrapy/scrapy#7166 · 1 comment · Opened Dec 1, 2025
web-poet: check that @field methods match returned item class fields Open

new rule

Difficulty 3/5 1-2 days Newbie friendliness 45/100

scrapy/scrapy-lint#110 · Opened Sep 15, 2025
Support `pyproject.toml` as a preferred alternative to `scrapy.cfg` Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 30/100

scrapy/scrapy#7030 · 1 comment · 1 reaction · Opened Aug 28, 2025
Discourage using `CrawlSpider` Open

discuss docs enhancement

Difficulty 1/5 Under an hour Newbie friendliness 45/100

scrapy/scrapy#7025 · 4 comments · Opened Aug 23, 2025
Add special treatment for ZYTE_API_RETRY_POLICY Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 35/100

scrapy/scrapy-lint#108 · Opened Aug 21, 2025
scrapy settings --get does not account for add-ons Open

Difficulty 3/5 1-2 days Newbie friendliness 48/100

scrapy/scrapy#7010 · 4 comments · 1 reaction · Opened Aug 18, 2025
Normalize multi-line string dedenting Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 38/100

scrapy/scrapy-lint#105 · Opened Aug 16, 2025
SCP24: Do not require packages without dependencies Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 45/100

scrapy/scrapy-lint#104 · Opened Aug 16, 2025
New rule: Behind deprecation window Open

new rule

Difficulty 3/5 1-2 days Newbie friendliness 48/100

scrapy/scrapy-lint#102 · 1 reaction · Opened Aug 15, 2025
scrapy-zyte-api session politeness Open

enhancement

Difficulty 2/5 1-3 hours Newbie friendliness 58/100

scrapy/scrapy-lint#101 · Opened Aug 15, 2025
New rule: Scrapy version mismatch Open

new rule

Difficulty 3/5 1-2 days Newbie friendliness 45/100

scrapy/scrapy-lint#99 · Opened Aug 15, 2025
Add an option to automatically save unknown settings to the list of known settings Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 20/100

scrapy/scrapy-lint#95 · Opened Aug 15, 2025
Support noqa comments in non-Python files Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 30/100

scrapy/scrapy-lint#94 · Opened Aug 15, 2025

Showing the newest 100

This page lists what was indexed most recently. Advanced filter has the whole inventory, narrowed by language, difficulty and how long a task takes.

Open advanced filter

scrapy

Showing the newest 100

Get new issues in your inbox