Bug: Redis connection pool exhaustion on /downloads/ endpoint under high concurrency
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 700
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 11
Description
Describe the bug
Under high concurrency conditions, particularly during new Python release events, the /downloads/ endpoint may experience Redis connection pool exhaustion. The application is configured with gunicorn.conf setting workers = 2, which creates a bottleneck when combined with synchronous Redis operations in the downloads view.
This manifests as increased tail latency and potential redis.exceptions.ConnectionError entries in application logs during traffic spikes.
To Reproduce
- Access https://www.python.org/downloads/ during a high-traffic event (e.g., new Python release announcement)
- Monitor response times and Redis connection metrics via application monitoring
- Observe elevated error rates and latency under sustained concurrent requests
Expected behavior
- Connection pooling should gracefully handle burst traffic without exhaustion
- Stale-while-revalidate caching pattern should serve cached download statistics when Redis is under pressure
- Celery tasks triggered from download views should be truly asynchronous, not blocking Redis operations on the main request thread
URL to the issue
https://www.python.org/downloads/
Screenshots
N/A (requires production monitoring access: Datadog dashboards, Sentry error logs, or Redis connection pool metrics)
Browsers
Microsoft Edge
Operating System
Windows
Browser Version
N/A
Relevant log output
# Example of expected log entry during pool exhaustion (needs verification in production):
redis.exceptions.ConnectionError: Connection pool exhausted. Unable to acquire connection within timeout.
# Relevant files for investigation:
# - gunicorn.conf (workers = 2)
# - apps/downloads/views.py
# - config/settings/cache.py
Additional context
Recent PR #2764 added a sitetree cache purge command to the Makefile, indicating the team's awareness of caching best practices. Similar patterns should be applied to high-traffic endpoints like /downloads/. Django cache framework documentation recommends connection pooling for production deployments. The project's dependency modernization efforts (e.g., pygments bump in PR #2977) suggest the team is actively maintaining performance-critical components.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing gunicorn.conf, apps/downloads/views.py, and config/settings/cache.py, then inspect concurrency behavior using the application's Redis connection metrics and logs. Verify whether the downloads endpoint exhausts its pool and blocks on Redis or Celery work. Done means burst traffic no longer causes the reported exhaustion, latency, or connection errors, with cached statistics available under Redis pressure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, redis
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100