canonical / canonical/testflinger
Sporadic tracebacks when attempting to submit a job
- Dominant language
- Python
- Stars
- 26
- Forks
- 49
- Avg merge
- 4d 8h
- Merged PRs (30d)
- 16
Description
I've recently encountered the following traceback when attempting to submit jobs:
```
$ testflinger-cli submit -p aitken.yaml
Traceback (most recent call last):
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/urllib3/connectionpool.py", line 534, in _make_request
response = conn.getresponse()
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/urllib3/connection.py", line 516, in getresponse
httplib_response = super().getresponse()
File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
response.begin()
File "/usr/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.10/ssl.py", line 1303, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.10/ssl.py", line 1159, in read
return self._sslobj.read(len, buffer)
TimeoutError: The read operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/requests/adapters.py", line 667, in send
resp = conn.urlopen(
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/urllib3/util/retry.py", line 474, in increment
raise reraise(type(error), error, _stacktrace)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/urllib3/util/util.py", line 39, in reraise
raise value
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/urllib3/connectionpool.py", line 367, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='testflinger.canonical.com', port=443): Read timed out. (read timeout=15)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/snap/testflinger-cli/201/bin/testflinger-cli", line 8, in
sys.exit(cli())
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/testflinger_cli/__init__.py", line 55, in cli
tfcli.run()
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/testflinger_cli/__init__.py", line 180, in run
sys.exit(self.args.func())
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/testflinger_cli/__init__.py", line 538, in submit
self.check_online_agents_available(queue)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/testflinger_cli/__init__.py", line 574, in check_online_agents_available
agents = self.client.get_agents_on_queue(queue)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/testflinger_cli/client.py", line 294, in get_agents_on_queue
data = self.get(endpoint)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/testflinger_cli/client.py", line 59, in get
req = requests.get(uri, timeout=timeout, headers=headers)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/requests/adapters.py", line 713, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='testflinger.canonical.com', port=443): Read timed out. (read timeout=15)
```
I've seen this after resetting offline Testflinger agents, and now also after submitting a new job just after releasing a node that I'd been testing via a direct (non-Testflinger) MAAS deployment. The job runs normally after one (or, in the latest case, two) failures. In the specific case just shown, here's the aitken.yaml file I was using:
```
job_queue: aitken
provision_data:
distro: noble
reserve_data:
ssh_keys:
- lp:bladernr
- lp:rodsmith
timeout: 21600
```
My Testflinger snap information:
```
$ snap info testflinger-cli
name: testflinger-cli
summary: Testflinger CLI
publisher: Canonical Certification Team (ce-certification-qa)
store-url: https://snapcraft.io/testflinger-cli
contact: https://github.com/canonical/testflinger/issues
license: GPL-3.0-only
description: |
The testflinger-cli tool is used for interacting with the testflinger
server for submitting test jobs, checking status, getting results, and
streaming output.
commands:
- testflinger-cli
snap-id: O4E2kMeKVNOoXBW7r8IeiAHk2Un4UbfM
tracking: latest/stable
refresh-date: 14 days ago, at 17:43 CEST
channels:
latest/stable: 20250408 2025-04-10 (201) 6MB -
latest/candidate: 20250408 2025-04-10 (201) 6MB -
latest/beta: 20250428 2025-04-28 (205) 569kB -
latest/edge: 20250428 2025-04-28 (205) 569kB -
installed: 20250408 (201) 6MB -
```
The host on which I'm running `testflinger-cli` is running Ubuntu 25.04.
Contributor guide
Research direction
Start with testflinger_cli/__init__.py at submit() and check_online_agents_available(), then follow client.py through get_agents_on_queue() and get(). Reproduce the submit scenario with a transient server timeout and inspect how the request failure is surfaced. Done means intermittent submission failures are handled as intended without exposing the shown traceback, with coverage for the behavior if existing tests support it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, cli, networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100