Occasional `Connection reset by peer` by content app
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 598
- Forks
- 168
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 86
Description
Occasionally the pulp-content-app will return a 500 error due to a Connection reset by peer error. Subsequent requests complete successfully.
It has been observed that once this occurs it will occur repeatedly albeit infrequently until the pod is deleted/restarted.
Additionally the error typically occurs in only one of the three pulp-content-app replicas at a time.
[2022-02-24 19:10:18 +0000] [10] [ERROR] Error handling request
Traceback (most recent call last):
File "/venv/lib64/python3.8/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
resp = await request_handler(request)
File "/venv/lib64/python3.8/site-packages/aiohttp/web_app.py", line 504, in _handle
resp = await handler(request)
File "/venv/lib64/python3.8/site-packages/aiohttp/web_middlewares.py", line 117, in impl
return await handler(request)
File "/venv/lib64/python3.8/site-packages/pulpcore/content/authentication.py", line 41, in authenticate
return await handler(request)
File "/venv/lib64/python3.8/site-packages/pulpcore/cache/cache.py", line 212, in cached_function
bk = await self.default_base_key(request, self)
File "/venv/lib64/python3.8/site-packages/pulpcore/content/handler.py", line 156, in find_base_path_cached
index_p1 = await cached.exists(base_key=multiplied_base_paths)
File "/venv/lib64/python3.8/site-packages/pulpcore/cache/cache.py", line 50, in wrapper
return await func(*args, **kwargs)
File "/venv/lib64/python3.8/site-packages/pulpcore/cache/cache.py", line 153, in exists
return await self.redis.exists(*base_key)
File "/venv/lib64/python3.8/site-packages/aioredis/client.py", line 1061, in execute_command
conn = self.connection or await pool.get_connection(command_name, **options)
File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 1365, in get_connection
if await connection.can_read():
File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 848, in can_read
return await self._parser.can_read(timeout)
File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 362, in can_read
return self._buffer and bool(await self._buffer.can_read(timeout))
File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 274, in can_read
return bool(self.length) or await self._read_from_socket(
File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 247, in _read_from_socket
data = await self._stream.read(self.socket_read_size)
File "/usr/lib64/python3.8/asyncio/streams.py", line 665, in read
raise self._exception
File "/usr/lib64/python3.8/asyncio/selector_events.py", line 848, in _read_ready__data_received
data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
2022-02-24 19:10:18,043 INFO gunicorn.access: X.X.X.X [24/Feb/2022:19:10:18 +0000] "GET /api/automation-hub/v3/artifacts/collections/published/ansible-tower-3.8.5.tar.gz?validate_token=539ad80e030608fa60520eb76a9553050727b4a2b95889b7de48ab81553be496:56be102481103ad85a7d0c4ec8b762a86767cdd21e84186e717b8419966014cb HTTP/1.1" 500 244 "-" "python-requests/2.26.0"
pulpcore==3.17.4
aioredis==2.0.0
AWS Elasticache for Redis: 5.0.5
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the traceback through pulpcore/content/authentication.py, pulpcore/cache/cache.py, and pulpcore/content/handler.py, focusing on the aioredis Redis call and the listed pulpcore, aioredis, and ElastiCache versions. Reproduce or characterize the reset against the content-app replicas and Redis setup. Done would require identifying the cause of the intermittent 500 response and confirming that the affected request path no longer fails.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, redis
- Domain
- backend, cloud, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100