Timeout trying to upload qcamera files
- Dominant language
- Python
- Stars
- 63.7k
- Forks
- 11.4k
- Avg merge
- 17h 53m
- Merged PRs (30d)
- 171
Description
### Describe the bug
When uploader tries to upload certain `qcamera.ts` files, it experiences a `ReadTimeout` from the Azure bucket. The file is actually uploaded successfully, since it does show up (and I can download it from) useradmin.
Uploader keeps retrying the same file until you remove it.
Never saw issues with qlogs (which go through the same process and go to the same bucket) or other uploads like bootlogs.
Exception:
```
ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='commadata2.blob.core.windows.net', port=443): Read timed out. (read timeout=10)")),
Traceback (most recent call last):
File "/usr/local/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 536, in _make_request
response = conn.getresponse()
^^^^^^^^^^^^^^^^^^
File "/usr/local/venv/lib/python3.12/site-packages/urllib3/connection.py", line 507, in getresponse
httplib_response = super().getresponse()
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse
response.begin()
File "/usr/lib/python3.12/http/client.py", line 331, in begin
version, status, reason = self._read_status()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/http/client.py", line 292, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/socket.py", line 707, in readinto
return self._sock.recv_into(b)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/ssl.py", line 1252, in recv_into
return self.read(nbytes, buffer)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/ssl.py", line 1104, in read
return self._sslobj.read(len, buffer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: The read operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/venv/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/usr/local/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 843, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/usr/local/venv/lib/python3.12/site-packages/urllib3/util/retry.py", line 474, in increment
raise reraise(type(error), error, _stacktrace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/venv/lib/python3.12/site-packages/urllib3/util/util.py", line 39, in reraise
raise value
File "/usr/local/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 789, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 538, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/local/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 369, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='commadata2.blob.core.windows.net', port=443): Read timed out. (read timeout=10)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/openpilot/system/loggerd/uploader.py", line 186, in upload
stat = self.do_upload(key, fn)
^^^^^^^^^^^^^^^^^^^^^^^
File "/data/openpilot/system/loggerd/uploader.py", line 159, in do_upload
response = requests.put(url, data=stream, headers=headers, timeout=10)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/venv/lib/python3.12/site-packages/requests/api.py", line 130, in put
return request("put", url, data=data, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/venv/lib/python3.12/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/venv/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/venv/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/venv/lib/python3.12/site-packages/requests/adapters.py", line 713, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='commadata2.blob.core.windows.net', port=443): Read timed out. (read timeout=10)
```
### Provide a route where the issue occurs
N/A
### openpilot version
0.9.9
### Additional info
_No response_
Contributor guide
Research direction
Start in system/loggerd/uploader.py at upload and do_upload, identified at lines 186 and 159 in the traceback. Inspect how requests.put timeout exceptions are handled and how completed files are retried, comparing qcamera.ts behavior with qlogs and bootlogs. Done means a successfully stored qcamera file is not repeatedly retried after a ReadTimeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100