hexojs / hexojs/hexo

The proposal of using Multi Threads

Open
#4,355 15 comments 12 reactions 0 assignees View on GitHub
#perfmatters help wanted proposal
Dominant language
TypeScript
Stars
41.8k
Forks
41
PR merge metrics
No merged PRs in 30d

Description

Since #550, the original creator of Hexo, @tommy351 want to speed up Hexo with multi core rendering. However, the #550 is never continued due to the difficulties of managing multiple Hexo instance.

Recently I have brought up Node.js `worker_threads` for a project (https://github.com/OI-wiki/OI-wiki/pull/2288) and learned something about `worker_threads`. With Node.js add support for `worker_threads`, it is now possible to bring up multi core rendering for Hexo again.

**Limit**

Workers Thread is designed to run CPU intensive tasks with simple algorism:

```
Independent Input => Workers Calculating => Independent Output
```

Thus we cannot run many difficult functions inside workers.

**Design**

As creating workers and destroy workers is still expensive (worker_threads are required to contact with main_thread), we should only create limited number of `worker_threads` (In https://github.com/OI-wiki/OI-wiki/pull/2288 I use the length of CPU Threads). Thus, a `WorkerPool` util should be made.

The `WorkPool` is designed to queue the task, manage task and make sure next task would run in an idle worker, thus it should have those method:

- `init()`: Init a worker pool with the queue (the queue could be an array). This will be called in `constructor`.
- `run(input)`: add a task to the queue, with `input` passed to the workers. A `Promise` will be returned (the result could be retrieved by `const output = await workerPool.run(input)`).
- `destroy()`: after all tasks is finished, destroy all the worker_threads created.

And here is an example about how to use `WorkPool`:

```js
// index.js
const { join } = require('path');
const { WorkerPool } = require('hexo-util');

const workerPath = join(__dirname + '/some_worker.js');
const cpuNums = require('os').cpus().length;

const pool = new WorkerPool(workerPath, cpuNums);

const tasksList = /* some stuff goes here ... */
const result = {};

Promise.all(tasksList.map(async task => {
const output = await pool.run(task);

// do something with output, maybe writeFile or push to a resultArray.
result[taskId] = output;
}).then(() => {
pool.destroy();

// do something with result object.
});
```

```js
// some_worker.js
const { isMainThread, parentPort } = require('worker_threads');

if (isMainThread) {
throw new Error('It is not a worker, it seems like a Main Thread');
}

async function job(input) {
// some stuff...
return output;
}

parentPort.on('message', async input => {
const output = await job(input);
parentPort.postMessage(output);
});
```

As you can see, the example I given is suitable for some of filters (likes `meta_generator`, `backtick_code_filter`) that we pass input to the filter and get output from it. But for more complicated job (like post rendering & template rendering) workers_thread still can't help.

cc @hexojs/core @tommy351

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.