cobrateam / cobrateam/splinter
HTTP Error 403: request disallowed by robots.txt
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 511
- PR merge metrics
- No merged PRs in 30d
Description
Running in python venv (MacOS) `Python 2.7.16`
Installed packages in venv:
```
splinter==0.11.0
zope.testbrowser==5.3.3
lxml==4.3.4
cssselect==1.0.3
mechanize==0.4.2
```
```python
from splinter import Browser
jenkins_url = 'some value here'
browser = Browser('zope.testbrowser')
browser.visit(jenkins_url)
```
getting back:
```
Traceback (most recent call last):
File "jenkins_test.py", line 9, in
browser.visit(jenkins_url)
File "/Users/homedir/Documents/Jenkins_Test/venv/lib/python2.7/site-packages/splinter/driver/zopetestbrowser.py", line 81, in visit
self._browser.open(url)
File "/Users/homedir/Documents/Jenkins_Test/venv/lib/python2.7/site-packages/zope/testbrowser/browser.py", line 252, in open
self._processRequest(url, make_request)
File "/Users/homedir/Documents/Jenkins_Test/venv/lib/python2.7/site-packages/zope/testbrowser/browser.py", line 276, in _processRequest
resp = make_request(reqargs)
File "/Users/homedir/Documents/Jenkins_Test/venv/lib/python2.7/site-packages/zope/testbrowser/browser.py", line 250, in make_request
return self.testapp.get(url, **args)
File "/Users/homedir/Documents/Jenkins_Test/venv/lib/python2.7/site-packages/webtest/app.py", line 335, in get
expect_errors=expect_errors)
File "/Users/homedir/Documents/Jenkins_Test/venv/lib/python2.7/site-packages/zope/testbrowser/browser.py", line 92, in do_request
self._assertAllowed(req.url)
File "/Users/homedir/Documents/Jenkins_Test/venv/lib/python2.7/site-packages/zope/testbrowser/browser.py", line 89, in _assertAllowed
raise RobotExclusionError(url, 403, msg, [], None)
zope.testbrowser.browser.RobotExclusionError: HTTP Error 403: request disallowed by robots.txt
```
What am I doing wrong ?
Tried to follow this docs here https://splinter.readthedocs.io/en/latest/drivers/zope.testbrowser.html
but ` ignore_robots=True` doesn't help:
```
Traceback (most recent call last):
File "jenkins_test.py", line 8, in
browser = Browser('zope.testbrowser', ignore_robots=True)
File "/Users/homedir/Documents/Jenkins_Test/venv/lib/python2.7/site-packages/splinter/browser.py", line 64, in Browser
return driver(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'ignore_robots'
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading splinter/driver/zopetestbrowser.py alongside the linked zope.testbrowser driver documentation, focusing on Browser construction and visit handling. Reproduce the Python 2.7 example and compare the documented ignore_robots option with the constructor behavior; done means the reported robots.txt failure and option mismatch are covered by an appropriate regression check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100