kernelci / kernelci/kernelci-pipeline

Sudden connection error in pipeline services

Open
#506 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
11
Forks
40
Avg merge
2d 13h
Merged PRs (30d)
14

Description

After starting API and Pipeline services, the services worked fine for some time. Then suddenly `monitor`, `tarball`, and `scheduler-k8s` services stopped. Other pipeline and API services were running OK while this issue was observed.

Error logs:
```
today at 10:13:0403/27/2024 04:43:04 AM UTC [ERROR] Traceback (most recent call last):
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn
today at 10:13:04 sock = connection.create_connection(
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 60, in create_connection
today at 10:13:04 for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/socket.py", line 962, in getaddrinfo
today at 10:13:04 for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 10:13:04socket.gaierror: [Errno -5] No address associated with hostname
today at 10:13:04
today at 10:13:04The above exception was the direct cause of the following exception:
today at 10:13:04
today at 10:13:04Traceback (most recent call last):
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
today at 10:13:04 response = self._make_request(
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 491, in _make_request
today at 10:13:04 raise new_e
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
today at 10:13:04 self._validate_conn(conn)
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
today at 10:13:04 conn.connect()
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 616, in connect
today at 10:13:04 self.sock = sock = self._new_conn()
today at 10:13:04 ^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 205, in _new_conn
today at 10:13:04 raise NameResolutionError(self.host, self, e) from e
today at 10:13:04urllib3.exceptions.NameResolutionError: : Failed to resolve 'staging.kernelci.org' ([Errno -5] No address associated with hostname)
today at 10:13:04
today at 10:13:04The above exception was the direct cause of the following exception:
today at 10:13:04
today at 10:13:04Traceback (most recent call last):
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
today at 10:13:04 resp = conn.urlopen(
today at 10:13:04 ^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
today at 10:13:04 retries = retries.increment(
today at 10:13:04 ^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment
today at 10:13:04 raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 10:13:04urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/listen/18845 (Caused by NameResolutionError(": Failed to resolve 'staging.kernelci.org' ([Errno -5] No address associated with hostname)"))
today at 10:13:04
today at 10:13:04During handling of the above exception, another exception occurred:
today at 10:13:04
today at 10:13:04Traceback (most recent call last):
today at 10:13:04 File "/home/kernelci/pipeline/base.py", line 69, in run
today at 10:13:04 status = self._run(context)
today at 10:13:04 ^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/home/kernelci/./pipeline/monitor.py", line 60, in _run
today at 10:13:04 event = self._api.receive_event(sub_id)
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/kernelci/api/latest.py", line 138, in receive_event
today at 10:13:04 resp = self._get(path)
today at 10:13:04 ^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/kernelci/api/__init__.py", line 66, in _get
today at 10:13:04 resp = requests.get(
today at 10:13:04 ^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 73, in get
today at 10:13:04 return request("get", url, params=params, **kwargs)
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
today at 10:13:04 return session.request(method=method, url=url, **kwargs)
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
today at 10:13:04 resp = self.send(prep, **send_kwargs)
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
today at 10:13:04 r = adapter.send(request, **kwargs)
today at 10:13:04 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 10:13:04 File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 519, in send
today at 10:13:04 raise ConnectionError(e, request=request)
today at 10:13:04requests.exceptions.ConnectionError: HTTPSConnectionPool(host='staging.kernelci.org', port=9000): Max retries exceeded with url: /latest/listen/18845 (Caused by NameResolutionError(": Failed to resolve 'staging.kernelci.org' ([Errno -5] No address associated with hostname)"))
today at 10:13:04
today at 10:24:09Container stopped
```

It seems like something is blocking the pipeline services from accessing API. Maybe some Sysadmin related issue? @nuclearcat

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with pipeline/monitor.py at _run and pipeline/base.py at run, then trace the request through kernelci/api/latest.py receive_event and kernelci/api/__init__.py _get. Reproduce or investigate the staging.kernelci.org resolution failure and determine why monitor, tarball, and scheduler-k8s stop. Done means the root cause is identified and the affected services handle the API connection failure appropriately.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.