pulp-worker creates some database pressure when scaling beyond 250 workers
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 598
- Forks
- 168
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 86
Description
Version
3.78.0
Describe the bug
We ran a test to check the number of idle pulp-workers that a system could handle. We tested it with 50, 100, 250, 500 and 1000 workers.
The database instance is a AWS RDS Postgres, running on a db.m7g.2xlarge machine, with 8 Cores and 32GiB of RAM.
Each pulp-worker have access to one entire core, with 512MiB of RAM, which is not used most of the time.
The telemetry was disabled for this test(OTEL_ENABLED=False).
Each run consisted in scale the number of workers in a certain pace until it reaches the desired number and stay stable for 1 hour.
The test with 50 and 100 where perfect without issues. From 250 to 1000 we started to see offline workers from time to time.
When running with 250 workers, less than 5% of workers got offline from time to time. The system took less than 5 minutes to recover to a stable state with all 250 workers online for at least 30 minutes.
When running with 500 workers, less than 15% of workers got offline from time to time. The system took less than 5 minutes to recover to a stable state with all 500 workers online for at least 30 minutes.
When running with 1000 workers, 90% of workers for offline from time to time. The system usually took no less than 30 minutes to recover all 1000 workers, and usually it took lots of opened sessions, maxing out the db's CPU, as we can see in the following picture:
We can see that during some time, the CPU was the most demanded resource from the database, with some queries topping its usage:
In the above picture, we can see that the query SELECT pg_try_advisory_lock(?) was the most called query during the test, and it demanded most of CPU time.
To Reproduce
Scale up the number of pulp-workers to 1000, with steps of 100 workers per time.
Expected behavior
We expected the database to have a low number of opened sessions and CPU running around 25% of usage.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported scale-up from 100 to 1000 pulp-workers and monitor PostgreSQL sessions and CPU on AWS RDS. Investigate the reported SELECT pg_try_advisory_lock(?) query and the worker offline/recovery behavior. Done means scaling no longer causes excessive sessions, CPU pressure, or widespread workers going offline.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, postgresql, python
- Domain
- backend, databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100