cobrateam / cobrateam/splinter

HTTP Error 403: request disallowed by robots.txt

Open
#702 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2.7k
Forks
511
PR merge metrics
No merged PRs in 30d

Description

Running in python venv (MacOS) `Python 2.7.16`

Installed packages in venv:
```
splinter==0.11.0
zope.testbrowser==5.3.3
lxml==4.3.4
cssselect==1.0.3
mechanize==0.4.2
```

```python
from splinter import Browser

jenkins_url = 'some value here'

browser = Browser('zope.testbrowser')
browser.visit(jenkins_url)
```

getting back:

```
Traceback (most recent call last):
File "jenkins_test.py", line 9, in
browser.visit(jenkins_url)
File "/Users/homedir/Documents/Jenkins_Test/venv/lib/python2.7/site-packages/splinter/driver/zopetestbrowser.py", line 81, in visit
self._browser.open(url)
File "/Users/homedir/Documents/Jenkins_Test/venv/lib/python2.7/site-packages/zope/testbrowser/browser.py", line 252, in open
self._processRequest(url, make_request)
File "/Users/homedir/Documents/Jenkins_Test/venv/lib/python2.7/site-packages/zope/testbrowser/browser.py", line 276, in _processRequest
resp = make_request(reqargs)
File "/Users/homedir/Documents/Jenkins_Test/venv/lib/python2.7/site-packages/zope/testbrowser/browser.py", line 250, in make_request
return self.testapp.get(url, **args)
File "/Users/homedir/Documents/Jenkins_Test/venv/lib/python2.7/site-packages/webtest/app.py", line 335, in get
expect_errors=expect_errors)
File "/Users/homedir/Documents/Jenkins_Test/venv/lib/python2.7/site-packages/zope/testbrowser/browser.py", line 92, in do_request
self._assertAllowed(req.url)
File "/Users/homedir/Documents/Jenkins_Test/venv/lib/python2.7/site-packages/zope/testbrowser/browser.py", line 89, in _assertAllowed
raise RobotExclusionError(url, 403, msg, [], None)
zope.testbrowser.browser.RobotExclusionError: HTTP Error 403: request disallowed by robots.txt
```

What am I doing wrong ?

Tried to follow this docs here https://splinter.readthedocs.io/en/latest/drivers/zope.testbrowser.html
but ` ignore_robots=True` doesn't help:

```
Traceback (most recent call last):
File "jenkins_test.py", line 8, in
browser = Browser('zope.testbrowser', ignore_robots=True)
File "/Users/homedir/Documents/Jenkins_Test/venv/lib/python2.7/site-packages/splinter/browser.py", line 64, in Browser
return driver(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'ignore_robots'
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading splinter/driver/zopetestbrowser.py alongside the linked zope.testbrowser driver documentation, focusing on Browser construction and visit handling. Reproduce the Python 2.7 example and compare the documented ignore_robots option with the constructor behavior; done means the reported robots.txt failure and option mismatch are covered by an appropriate regression check.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.