pulp / pulp/pulpcore

Occasional `Connection reset by peer` by content app

Open
#2,334 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

GalaxyNG Issue
Dominant language
Python
Stars
598
Forks
168
Avg merge
1d 4h
Merged PRs (30d)
86

Description

Occasionally the pulp-content-app will return a 500 error due to a Connection reset by peer error. Subsequent requests complete successfully.

It has been observed that once this occurs it will occur repeatedly albeit infrequently until the pod is deleted/restarted.
Additionally the error typically occurs in only one of the three pulp-content-app replicas at a time.

[2022-02-24 19:10:18 +0000] [10] [ERROR] Error handling request
Traceback (most recent call last):
  File "/venv/lib64/python3.8/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
    resp = await request_handler(request)
  File "/venv/lib64/python3.8/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/venv/lib64/python3.8/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/venv/lib64/python3.8/site-packages/pulpcore/content/authentication.py", line 41, in authenticate
    return await handler(request)
  File "/venv/lib64/python3.8/site-packages/pulpcore/cache/cache.py", line 212, in cached_function
    bk = await self.default_base_key(request, self)
  File "/venv/lib64/python3.8/site-packages/pulpcore/content/handler.py", line 156, in find_base_path_cached
    index_p1 = await cached.exists(base_key=multiplied_base_paths)
  File "/venv/lib64/python3.8/site-packages/pulpcore/cache/cache.py", line 50, in wrapper
    return await func(*args, **kwargs)
  File "/venv/lib64/python3.8/site-packages/pulpcore/cache/cache.py", line 153, in exists
    return await self.redis.exists(*base_key)
  File "/venv/lib64/python3.8/site-packages/aioredis/client.py", line 1061, in execute_command
    conn = self.connection or await pool.get_connection(command_name, **options)
  File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 1365, in get_connection
    if await connection.can_read():
  File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 848, in can_read
    return await self._parser.can_read(timeout)
  File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 362, in can_read
    return self._buffer and bool(await self._buffer.can_read(timeout))
  File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 274, in can_read
    return bool(self.length) or await self._read_from_socket(
  File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 247, in _read_from_socket
    data = await self._stream.read(self.socket_read_size)
  File "/usr/lib64/python3.8/asyncio/streams.py", line 665, in read
    raise self._exception
  File "/usr/lib64/python3.8/asyncio/selector_events.py", line 848, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
2022-02-24 19:10:18,043 INFO gunicorn.access: X.X.X.X [24/Feb/2022:19:10:18 +0000] "GET /api/automation-hub/v3/artifacts/collections/published/ansible-tower-3.8.5.tar.gz?validate_token=539ad80e030608fa60520eb76a9553050727b4a2b95889b7de48ab81553be496:56be102481103ad85a7d0c4ec8b762a86767cdd21e84186e717b8419966014cb HTTP/1.1" 500 244 "-" "python-requests/2.26.0" 

pulpcore==3.17.4
aioredis==2.0.0
AWS Elasticache for Redis: 5.0.5

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the traceback through pulpcore/content/authentication.py, pulpcore/cache/cache.py, and pulpcore/content/handler.py, focusing on the aioredis Redis call and the listed pulpcore, aioredis, and ElastiCache versions. Reproduce or characterize the reset against the content-app replicas and Redis setup. Done would require identifying the cause of the intermittent 500 response and confirming that the affected request path no longer fails.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, redis
Domain
backend, cloud, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.