couchbaselabs / couchbaselabs/promtimer

Promtimer should support alternate address (was: promtimer issue working with remote AWS instances)

Open
#50 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
9
Forks
26
Avg merge
11d 1h
Merged PRs (30d)
2

Description

I was encountering an issue when starting promtimer against a remote AWS instance. I put a printf in urllib/request.py in do_open
```1346 try:
1347 try:
1348 print(f"{req.get_method()} {req.selector} {req.data} {headers}")
1349 h.request(req.get_method(), req.selector, req.data, headers,
1350 encode_chunked=req.has_header('Transfer-encoding'))
1351 except OSError as err: # timeout error
1352 raise URLError(err)
1353 r = h.getresponse()
```
and see this crash. Note the /pools/default/buckets is being done against the localhost (127.0.0.1) which is my mac. This is a single node, just created cluster so maybe that's a reason
```
steve.watanabe @ ~/promtimer/bin (master) $ ./promtimer --cluster ec2-52-23-193-116.compute-1.amazonaws.com:8091 --user Administrator --password asdasd
GET /pools/default/nodeServices None {'Host': 'ec2-52-23-193-116.compute-1.amazonaws.com:8091', 'User-Agent': 'Python-urllib/3.11', 'Connection': 'close'}
GET /pools/default/nodeServices None {'Host': 'ec2-52-23-193-116.compute-1.amazonaws.com:8091', 'User-Agent': 'Python-urllib/3.11', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Connection': 'close'}
GET /pools/default/buckets None {'Host': '127.0.0.1:8091', 'User-Agent': 'Python-urllib/3.11', 'Connection': 'close'}
Traceback (most recent call last):
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1349, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1037, in _send_output
self.send(msg)
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 975, in send
self.connect()
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 941, in connect
self.sock = self._create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/socket.py", line 851, in create_connection
raise exceptions[0]
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/socket.py", line 836, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/steve.watanabe/promtimer/bin/../promtimer/promtimer.py", line 336, in
main()
File "/Users/steve.watanabe/promtimer/bin/../promtimer/promtimer.py", line 302, in main
buckets = stats_sources[0].get_buckets()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/steve.watanabe/promtimer/promtimer/cbstats.py", line 412, in get_buckets
response = util.execute_request('{}:{}'.format(self.host(), self.port()),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/steve.watanabe/promtimer/promtimer/util.py", line 139, in execute_request
response = opener.open(request)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 519, in open
response = self._open(req, data)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 536, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 496, in _call_chain
result = func(*args)
^^^^^^^^^^^
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1378, in http_open
return self.do_open(http.client.HTTPConnection, req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1352, in do_open
raise URLError(err)
urllib.error.URLError:
```

I then add a second node to the cluster and try again. Note it uses a non-local IP address (172.31.15.151) but not the external one and so the connection times out.

```
steve.watanabe @ ~/promtimer/bin (master) $ ./promtimer --cluster ec2-52-23-193-116.compute-1.amazonaws.com:8091 --user Administrator --password asdasd
GET /pools/default/nodeServices None {'Host': 'ec2-52-23-193-116.compute-1.amazonaws.com:8091', 'User-Agent': 'Python-urllib/3.11', 'Connection': 'close'}
GET /pools/default/nodeServices None {'Host': 'ec2-52-23-193-116.compute-1.amazonaws.com:8091', 'User-Agent': 'Python-urllib/3.11', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Connection': 'close'}
GET /pools/default/buckets None {'Host': '172.31.15.151:8091', 'User-Agent': 'Python-urllib/3.11', 'Connection': 'close'}
Traceback (most recent call last):
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1349, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1037, in _send_output
self.send(msg)
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 975, in send
self.connect()
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 941, in connect
self.sock = self._create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/socket.py", line 851, in create_connection
raise exceptions[0]
File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/socket.py", line 836, in create_connection
sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out
```

I skimmed the promtimer code and see it is using /pools/default/nodeServices

Running on my node
```
"thisNode": true,
"hostname": "172.31.15.151"

"hostname": "ec2-54-88-70-0.compute-1.amazonaws.com"
```
and
```
"hostname": "172.31.15.151"

"thisNode": true,
"hostname": "ec2-54-88-70-0.compute-1.amazonaws.com"
```
Having entered this issue perhaps there's two issues: 1) remote promtimer doesn't work against a node that doesn't have a "hostname" in the results from /pools/default/nodeServices and 2) ns_server may not be returning the right hostname...though I'm not sure how ns_server would know the IP isn't the external IP.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.