kernelci / kernelci/kernelci-pipeline
Sudden connection error in pipeline services
- Dominant language
- Python
- Stars
- 11
- Forks
- 40
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 14
Description
After starting API and Pipeline services, the services worked fine for some time. Then suddenly `monitor`, `tarball`, and `scheduler-k8s` services stopped. Other pipeline and API services were running OK while this issue was observed.
Error logs:
```
today at 10:13:0403/27/2024 04:43:04 AM UTC [ERROR] Traceback (most recent call last):
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn
today at 10:13:04 sock = connection.create_connection(
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 60, in create_connection
today at 10:13:04 for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/socket.py", line 962, in getaddrinfo
today at 10:13:04 for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 10:13:04socket.gaierror: [Errno -5] No address associated with hostname
today at 10:13:04
today at 10:13:04The above exception was the direct cause of the following exception:
today at 10:13:04
today at 10:13:04Traceback (most recent call last):
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
today at 10:13:04 response = self._make_request(
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 491, in _make_request
today at 10:13:04 raise new_e
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
today at 10:13:04 self._validate_conn(conn)
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
today at 10:13:04 conn.connect()
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 616, in connect
today at 10:13:04 self.sock = sock = self._new_conn()
today at 10:13:04 ^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 205, in _new_conn
today at 10:13:04 raise NameResolutionError(self.host, self, e) from e
today at 10:13:04urllib3.exceptions.NameResolutionError: : Failed to resolve 'staging.kernelci.org' ([Errno -5] No address associated with hostname)
today at 10:13:04
today at 10:13:04The above exception was the direct cause of the following exception:
today at 10:13:04
today at 10:13:04Traceback (most recent call last):
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
today at 10:13:04 resp = conn.urlopen(
today at 10:13:04 ^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
today at 10:13:04 retries = retries.increment(
today at 10:13:04 ^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment
today at 10:13:04 raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 10:13:04urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/listen/18845 (Caused by NameResolutionError(": Failed to resolve 'staging.kernelci.org' ([Errno -5] No address associated with hostname)"))
today at 10:13:04
today at 10:13:04During handling of the above exception, another exception occurred:
today at 10:13:04
today at 10:13:04Traceback (most recent call last):
today at 10:13:04 File "/home/kernelci/pipeline/base.py", line 69, in run
today at 10:13:04 status = self._run(context)
today at 10:13:04 ^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/home/kernelci/./pipeline/monitor.py", line 60, in _run
today at 10:13:04 event = self._api.receive_event(sub_id)
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/kernelci/api/latest.py", line 138, in receive_event
today at 10:13:04 resp = self._get(path)
today at 10:13:04 ^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/kernelci/api/__init__.py", line 66, in _get
today at 10:13:04 resp = requests.get(
today at 10:13:04 ^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 73, in get
today at 10:13:04 return request("get", url, params=params, **kwargs)
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
today at 10:13:04 return session.request(method=method, url=url, **kwargs)
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
today at 10:13:04 resp = self.send(prep, **send_kwargs)
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
today at 10:13:04 r = adapter.send(request, **kwargs)
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 519, in send
today at 10:13:04 raise ConnectionError(e, request=request)
today at 10:13:04requests.exceptions.ConnectionError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/listen/18845 (Caused by NameResolutionError(": Failed to resolve 'staging.kernelci.org' ([Errno -5] No address associated with hostname)"))
today at 10:13:04
today at 10:24:09Container stopped
```
It seems like something is blocking the pipeline services from accessing API. Maybe some Sysadmin related issue? @nuclearcat
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with pipeline/monitor.py at _run and pipeline/base.py at run, then trace the request through kernelci/api/latest.py receive_event and kernelci/api/__init__.py _get. Reproduce or investigate the staging.kernelci.org resolution failure and determine why monitor, tarball, and scheduler-k8s stop. Done means the root cause is identified and the affected services handle the API connection failure appropriately.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100