josdejong / josdejong/workerpool
Pool should use availableParallelism, not cpus, when determining maxWorkers
- Dominant language
- JavaScript
- Stars
- 2.3k
- Forks
- 164
- PR merge metrics
- No merged PRs in 30d
Description
When determining the default `maxWorkers`, `Pool` leverages the number of CPUs reported by `os.cpus()` (via the environment module). This is an incorrect use of `os.cpus()` and the `os.availableParallelism()` function is provided for that purpose. The documentation for `os.cpus()` explicitly states:
> os.cpus().length should not be used to calculate the amount of parallelism available to an application. Use [os.availableParallelism()](https://nodejs.org/api/os.html#osavailableparallelism) for this purpose.
Why does this matter? In environments where the application is running inside of containers -- in my case an Ember.js build running in Kubernetes -- the `os.cpus().length` call returns the number of CPUs on the host machine; not the number of CPUs available to the container. This causes more parallelism in the worker pool than is actually efficient. In my use case, it also results in an out of memory error.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating Pool's default maxWorkers calculation and the environment module that currently obtains the CPU count. Replace the CPU-count source with availableParallelism, then run the existing test suite and confirm the default worker count reflects parallelism available to the application, including container limits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100