josdejong / josdejong/workerpool

Concurrent async tasks on a single worker

Open
#371 1 comment 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
JavaScript
Stars
2.3k
Forks
164
PR merge metrics
No merged PRs in 30d

Description

Workerpool allows each thread to process only a single task at a time, which means that each task must be strictly CPU-bound, and any IO makes the pool work inefficiently.

At the same time, there are some traits of supporting multiple parallel tasks per threads in the code: https://github.com/josdejong/workerpool/blob/master/src/WorkerHandler.js#L332 - I think this is for the case when the thread is in "wait for ready" state only though, and due to this check: https://github.com/josdejong/workerpool/blob/master/src/Pool.js#L248 - it never schedules more than 1 task to a "busy" thread.

Piscina has a feature of scheduling multiple tasks to the same thread.

Are there any plans to implement the similar thing in workerpool (considering that the most of the infrastructure for this is already in the code)?

Repro of the serialization:

```bash
cat <<'EOT' > main.js
const workerpool = require("workerpool");
async function main() {
const pool = workerpool.pool(`${__dirname}/worker.js`, { minWorkers: 1, maxWorkers: 1 });
await Promise.all([
pool.exec("test", [1]),
pool.exec("test", [2]),
pool.exec("test", [3]),
pool.exec("test", [4])
]);
}
main();
EOT

cat <<'EOT' > worker.js
const workerpool = require("workerpool");
const delay = require("delay");
workerpool.worker({
test: async (v) => {
console.log(`[${v}] started`);
await delay(1000);
console.log(`[${v}] finished`);
}
});
EOT

node main.js

[1] started
[1] finished
[2] started
[2] finished
[3] started
[3] finished
[4] started
[4] finished
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.