urllib.robotparser.RobotFileParser.can_fetch crashes on a malformed URL
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 35.9k
- PR merge metrics
- PR metrics pending
Description
Bug description:
urllib.robotparser.RobotFileParser.can_fetch is documented to return a bool,
but it passes url straight to urllib.parse._urlsplit, so a malformed URL
(an unterminated IPv6 authority) raises a bare ValueError instead of a
result:
import urllib.robotparser
p = urllib.robotparser.RobotFileParser()
p.parse(["User-agent: *", "Disallow: /"])
p.can_fetch("*", "http://[::1") # ValueError: Invalid IPv6 URL
The url argument is crawl-frontier-controlled (it comes from parsed pages).
Proposed behavior: a URL that cannot be parsed matches no rule, so can_fetch
returns True (RFC 9309 default that access is not restricted when no rule
applies), rather than raising or being conservatively False.
This is a sibling of #153404 (robotparser raising ValueError on a non-decimal
digit in Crawl-delay / Request-rate); the sink there is the robots.txt
content, here it is the url argument.
CPython versions tested on:
3.13, 3.14, 3.15
Operating systems tested on:
macOS
Linked PRs
- gh-153796
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at urllib.robotparser.RobotFileParser.can_fetch and reproduce the malformed URL case with an unterminated IPv6 authority. Check how urllib.parse._urlsplit handles the input. Done means can_fetch returns True instead of raising ValueError, with coverage for the malformed URL behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100