Workers affected by V8 aborting on virtual allocation failure

• Version: 4b6e4c1eb110
• Platform: Linux 4.19.15 x86_64
• Subsystem: worker_threads

A process aborts when V8 fails to allocate virtual memory. This is detremental to using Workers because:

1. If a Worker fails to allocate virtual memory during initialization the process aborts.
2. V8's virtual memory allocator doesn't play nicely with ulimit.

To demonstrate this issue:

1. Set ulimit -v to some value smaller than actual physically available memory (I've allocated 8GB out of 16GB).
2. Run the following script that spawns a set number of Workers (the number to spawn may need adjusting depending on the system):

const WORKER_COUNT = 60; // Value may need adjustment

const { Worker, parentPort, workerData, threadId } = require('worker_threads');
const worker_array = [];

if (workerData) {
  process._rawDebug(workerData, threadId, process.pid, process.ppid);
  parentPort.postMessage(42);
  setTimeout(() => {}, 1000000);
  return;
}

(function runner(n) {
  if (++n > WORKER_COUNT) return setTimeout(killAll, 10);
  const w = new Worker(__filename, { workerData: n });
  w.on('message', () => {
    process._rawDebug(JSON.stringify(getMemUsage()));
    runner(n);
  });
  w.on('exit', c => process._rawDebug(`${n} exited with ${c}`));
  worker_array.push(w);
})(0);

function killAll() {
  for (let w of worker_array)
    w.terminate();
}

function getMemUsage() {
  const o = process.memoryUsage();
  for (let i in o) o[i] = o[i] / 1024;
  return o;
}

Which results in a Fatal process OOM in CodeRange setup: allocate virtual memory error. With the stack trace of:

 * thread #1: tid = 4094, 0x0000000003eab702 node_g`v8::base::OS::Abort() at platform-posix.cc:399, name = 'node_g', stop reason = signal SIGILL: illegal instruction operand
  * frame #0: 0x0000000003eab702 node_g`v8::base::OS::Abort() at platform-posix.cc:399
    frame #1: 0x0000000002954ac2 node_g`v8::Utils::ReportOOMFailure(isolate=0x0000000005c13170, location="CodeRange setup: allocate virtual memory", is_heap_oom=false) at api.cc:460
    frame #2: 0x000000000295492d node_g`v8::internal::V8::FatalProcessOutOfMemory(isolate=0x0000000005c13170, location="CodeRange setup: allocate virtual memory", is_heap_oom=false) at api.cc:428
    frame #3: 0x000000000322e4a6 node_g`v8::internal::MemoryAllocator::InitializeCodePageAllocator(this=0x0000000005c076d0, page_allocator=0x00000000045aa030, requested=134217728) a t spaces.cc:168
    frame #4: 0x000000000322e083 node_g`v8::internal::MemoryAllocator::MemoryAllocator(this=0x0000000005c076d0, isolate=0x0000000005c13170, capacity=1526909922, code_range_size=0) at spaces.cc:132
    frame #5: 0x000000000318d6da node_g`v8::internal::Heap::SetUp(this=0x0000000005c13190) at heap.cc:4349
    frame #6: 0x000000000331aa77 node_g`v8::internal::Isolate::Init(this=0x0000000005c13170, des=0x00007fffffff6db8) at isolate.cc:3176
    frame #7: 0x00000000036e558c node_g`v8::internal::Snapshot::Initialize(isolate=0x0000000005c13170) at snapshot-common.cc:55
    frame #8: 0x0000000002994b0d node_g`v8::Isolate::Initialize(isolate=0x0000000005c13170, params=0x00007fffffff7170) at api.cc:8224
    frame #9: 0x00000000025db63b node_g`node::NewIsolate(allocator=0x00007fff34002c80, event_loop=0x0000000005bf2530) at node.cc:1420
    frame #10: 0x000000000270d9a6 node_g`node::worker::Worker::Worker(this=0x0000000005bf2500, env=0x00007fffffffcd08, wrap=(val_ = 0x00007fffffff84a0), url="\xb0?"..., per_isolate_ opts=nullptr) at node_worker.cc:106

Examination of the strace log shows:

31103 mmap(0x129b53343000, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
31103 mmap(0x129b53343000, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
31103 write(2, "\n#\n# Fatal process OOM in CodeRa"..., 70) = 70

Though if I run the same script on a machine with 8GB physical memory and ulimit -v unlimited the script doesn't have a problem.

In the end, this is probably something that'd need to be resolved by V8. Both to allocate virtual memory more intelligently and allow the Isolate to notify when no more memory could be allocated. Then have that hooked into the Worker's 'error' event.

Reproduzieren Sie den Fehler mit dem bereitgestellten Worker-Skript und einem begrenzten ulimit -v. Beginnen Sie bei node_worker.cc:106 und verfolgen Sie den V8-Initialisierungs-Stack bis durch MemoryAllocator::InitializeCodePageAllocator; vergleichen Sie den Abort-Pfad mit dem angeforderten Worker-Verhalten 'error'. Als abgeschlossen würde eine definierte, getestete Reaktion auf einen Fehler bei der virtuellen Allokation anstelle eines Prozessabbruchs gelten.