ansible / ansible/awx-operator

ConnectionResetError: [Errno 104] Connection reset by peer

Open
#1,925 2 comments 0 reactions 0 assignees View on GitHub
community needs_triage
Dominant language
Jinja
Stars
1.5k
Forks
714
Avg merge
2d 3h
Merged PRs (30d)
3

Description

### Please confirm the following

- [X] I agree to follow this project's [code of conduct](https://docs.ansible.com/ansible/latest/community/code_of_conduct.html).
- [X] I have checked the [current issues](https://github.com/ansible/awx-operator/issues) for duplicates.
- [X] I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

### Bug Summary

Hello Everyone,

Suddenly getting below error while invoking API. This was working fine until 07/20, Saturday. Anyone facing same error?

Note: We are encountering this issue on all the AWX instances throughout the all locations.

AWX INFO *****************
cat kustomization.yaml
apiVersion: [kustomize.config.k8s.io/v1beta1](http://kustomize.config.k8s.io/v1beta1)
kind: Kustomization
resources:
# Find the latest tag here: https://github.com/ansible/awx-operator/releases
- [github.com/ansible/awx-operator/config/default?ref=0.29.0](http://github.com/ansible/awx-operator/config/default?ref=0.29.0)

- awx-pro.yaml
- my-ingress.yml

# Set the image tags to match the git version from above
images:
- name: [quay.io/ansible/awx-operator](http://quay.io/ansible/awx-operator)
newTag: 0.29.0

# Specify a custom namespace in which to install AWX
namespace: awx-lvn

Error Info ************************

Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
conn.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connection.py", line 411, in connect
self.sock = ssl_wrap_socket(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib64/python3.9/ssl.py", line 501, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib64/python3.9/ssl.py", line 1041, in _create
self.do_handshake()
File "/usr/lib64/python3.9/ssl.py", line 1310, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

### AWX Operator version

0.29.0

### AWX version

21.6.0

### Kubernetes platform

minikube

### Kubernetes/Platform version

Minikube: 1.27

### Modifications

no

### Steps to reproduce

job_launch_url="$AWX_HOST/api/v2/job_templates/$ext_tempt_id/launch/"
job_id=$(curl -k -H "Authorization: Bearer $AWX_TOKEN" -X POST "$job_launch_url")
ext_job_id=$(extract_value 'id' "$job_id")
echo "Job '$ext_job_id' launched from template '$TEMPLATE_NAME'."

Issue occurring after job is launched through API from the client, the job launches which hits AWX server but then connection is getting rejected when AWX is trying to connect client

Error:

Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
conn.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connection.py", line 411, in connect
self.sock = ssl_wrap_socket(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib64/python3.9/ssl.py", line 501, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib64/python3.9/ssl.py", line 1041, in _create
self.do_handshake()
File "/usr/lib64/python3.9/ssl.py", line 1310, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

### Expected results

AWX should launch job on the target client.

### Actual results

Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
conn.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/connection.py", line 411, in connect
self.sock = ssl_wrap_socket(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib64/python3.9/ssl.py", line 501, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib64/python3.9/ssl.py", line 1041, in _create
self.do_handshake()
File "/usr/lib64/python3.9/ssl.py", line 1310, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

### Additional information

_No response_

### Operator Logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the supplied kustomization.yaml and the curl job-launch command against the AWX Operator 0.29.0 deployment on minikube. Review the reported urllib3 SSL traceback and available operator logs; no source file or test is identified. Done means reproducing the reset, identifying its cause, and documenting a verified resolution or a clear project-side diagnosis.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.