python / python/pythondotorg

Bug: Redis connection pool exhaustion on /downloads/ endpoint under high concurrency

Open
#2,990 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
1.7k
Forks
700
Avg merge
2d 13h
Merged PRs (30d)
11

Description

Describe the bug

Under high concurrency conditions, particularly during new Python release events, the /downloads/ endpoint may experience Redis connection pool exhaustion. The application is configured with gunicorn.conf setting workers = 2, which creates a bottleneck when combined with synchronous Redis operations in the downloads view.

This manifests as increased tail latency and potential redis.exceptions.ConnectionError entries in application logs during traffic spikes.

To Reproduce
  1. Access https://www.python.org/downloads/ during a high-traffic event (e.g., new Python release announcement)
  2. Monitor response times and Redis connection metrics via application monitoring
  3. Observe elevated error rates and latency under sustained concurrent requests
Expected behavior
  • Connection pooling should gracefully handle burst traffic without exhaustion
  • Stale-while-revalidate caching pattern should serve cached download statistics when Redis is under pressure
  • Celery tasks triggered from download views should be truly asynchronous, not blocking Redis operations on the main request thread
URL to the issue

https://www.python.org/downloads/

Screenshots
N/A (requires production monitoring access: Datadog dashboards, Sentry error logs, or Redis connection pool metrics)
Browsers

Microsoft Edge

Operating System

Windows

Browser Version

N/A

Relevant log output
# Example of expected log entry during pool exhaustion (needs verification in production):
redis.exceptions.ConnectionError: Connection pool exhausted. Unable to acquire connection within timeout.
# Relevant files for investigation:
# - gunicorn.conf (workers = 2)
# - apps/downloads/views.py
# - config/settings/cache.py
Additional context

Recent PR #2764 added a sitetree cache purge command to the Makefile, indicating the team's awareness of caching best practices. Similar patterns should be applied to high-traffic endpoints like /downloads/. Django cache framework documentation recommends connection pooling for production deployments. The project's dependency modernization efforts (e.g., pygments bump in PR #2977) suggest the team is actively maintaining performance-critical components.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing gunicorn.conf, apps/downloads/views.py, and config/settings/cache.py, then inspect concurrency behavior using the application's Redis connection metrics and logs. Verify whether the downloads endpoint exhausts its pool and blocks on Redis or Celery work. Done means burst traffic no longer causes the reported exhaustion, latency, or connection errors, with cached statistics available under Redis pressure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, redis
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.