internetarchive / internetarchive/openlibrary

Investigate updating `gunicorn` worker counts

Open
#10,429 0 comments 0 reactions 0 assignees View on GitHub
Lead: @cdrini Needs: Breakdown Priority: 3 Theme: Performance Type: Feature Request
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

### Proposal

@cdrini points out that many of our `gunicorn` workers are sitting idle waiting for requests. As per [`gunicorn`'s guidance](https://docs.gunicorn.org/en/latest/design.html#how-many-workers), we may be using too many workers.

Let's investigate lower the number of `gunicorn` workers in our servers.

### Justification

Having too many idle workers can negatively affect site performance.

Adding more [statsD instrumentation](https://docs.gunicorn.org/en/latest/instrumentation.html) could help us determine if we should adjust the worker count, and by how much.

### Breakdown

#### Requirements Checklist

* [ ]

#### Related files

*

#### Stakeholders

*


#### Instructions for Contributors

Please [run these commands](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#working-on-your-branch) to ensure your repository is up to date **before** [creating a new branch](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#making-changes-and-creating-a-pull-request) to work on this issue and **each time after** pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by reviewing the current Gunicorn server configuration alongside Gunicorn's worker-count and statsD instrumentation guidance, then measure worker utilization and site performance. Done means a documented worker-count recommendation supported by measurements, with any needed instrumentation identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.