pulp / pulp/pulpcore

500 error when listing a lot of RPM packages

Open
#5,455 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Issue
Dominant language
Python
Stars
598
Forks
168
Avg merge
1d 4h
Merged PRs (30d)
86

Description

Version
3.25.3

Describe the bug

[2024-05-24 17:33:02 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:14)
[2024-05-24 17:33:03 +0000] [14] [ERROR] Error handling request /api/pulp/default/api/v3/content/rpm/packages/?offset=0&limit=1
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/sync.py", line 178, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/usr/local/lib/python3.9/site-packages/opentelemetry/instrumentation/wsgi/__init__.py", line 589, in __call__
    iterable = self.wsgi(environ, start_response)
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/utils.py", line 1711, in runner
    return sentry_patched_function(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/django/__init__.py", line 174, in sentry_patched_wsgi_handler
    return SentryWsgiMiddleware(bound_old_app, use_x_forwarded_for)(
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/wsgi.py", line 109, in __call__
    reraise(*_capture_exception())
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/utils.py", line 1640, in reraise
    raise value
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/wsgi.py", line 102, in __call__
    response = self.app(
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/wsgi.py", line 124, in __call__
    response = self.get_response(request)
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/django/__init__.py", line 462, in sentry_patched_get_response
    rv = old_get_response(self, request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 140, in get_response
    response = self._middleware_chain(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/django/middleware.py", line 169, in __call__
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django_guid/middleware.py", line 79, in middleware
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/django/middleware.py", line 169, in __call__
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/utils/deprecation.py", line 134, in __call__
    response = response or self.get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/django/middleware.py", line 169, in __call__
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/whitenoise/middleware.py", line 124, in __call__
    return self.get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/django/middleware.py", line 169, in __call__
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/utils/deprecation.py", line 134, in __call__
    response = response or self.get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/django/middleware.py", line 169, in __call__
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/utils/deprecation.py", line 134, in __call__
    response = response or self.get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/django/middleware.py", line 169, in __call__
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/utils/deprecation.py", line 134, in __call__
    response = response or self.get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/django/middleware.py", line 169, in __call__
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/utils/deprecation.py", line 134, in __call__
    response = response or self.get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/django/middleware.py", line 169, in __call__
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/utils/deprecation.py", line 134, in __call__
    response = response or self.get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/django/middleware.py", line 169, in __call__
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/utils/deprecation.py", line 134, in __call__
    response = response or self.get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/django/middleware.py", line 169, in __call__
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/middleware.py", line 24, in __call__
    response = self.get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/django/views.py", line 84, in sentry_wrapped_callback
    return callback(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/viewsets.py", line 124, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/mixins.py", line 40, in list
    page = self.paginate_queryset(queryset)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/generics.py", line 175, in paginate_queryset
    return self.paginator.paginate_queryset(queryset, self.request, view=self)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/pagination.py", line 395, in paginate_queryset
    self.count = self.get_count(queryset)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/pagination.py", line 530, in get_count
    return queryset.count()
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 608, in count
    return self.query.get_count(using=self.db)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/query.py", line 568, in get_count
    return obj.get_aggregation(using, {"__count": Count("*")})["__count"]
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/query.py", line 554, in get_aggregation
    result = compiler.execute_sql(SINGLE)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/utils.py", line 1711, in runner
    return sentry_patched_function(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/django/__init__.py", line 641, in execute
    result = real_execute(self, sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/psycopg/cursor.py", line 728, in execute
    self._conn.wait(
  File "/usr/local/lib/python3.9/site-packages/psycopg/connection.py", line 969, in wait
    return waiting.wait(gen, self.pgconn.socket, timeout=timeout)
  File "psycopg_binary/_psycopg/waiting.pyx", line 193, in psycopg_binary._psycopg.wait_c
  File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 203, in handle_abort
    sys.exit(1)
SystemExit: 1
('pulp [-]: ::ffff:10.131.8.12 - - [24/May/2024:17:33:03 +0000] "GET /api/pulp/default/api/v3/content/rpm/packages/?offset=0&limit=1 HTTP/1.1" 500 0 "-" "-"',)

To Reproduce
Load up pulp with many thousands of packages. Then run:

$ pulp rpm content list --limit 1
Error: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>
    
  </body>
</html>

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here. Please provide links to any previous discussions via Discourse or Bugzilla.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with pulp rpm content list --limit 1 against an installation containing many thousands of packages, then inspect the /api/pulp/default/api/v3/content/rpm/packages/ listing path and its pagination count step. Use the traceback to focus on the database count reached through Django REST Framework pagination. Done means the endpoint returns a valid package-list response instead of timing out with HTTP 500.

Written by the indexing model from the issue text.

Assessment

Tech stack
django, python
Domain
api, backend, databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.