Node.js consuming and reporting heap_size_limit larger than memory available via cgroups v2

Version

v26.3.1 (reproduced with latest Docker images for 20.x-26.x)

Platform

Linux dfcddcdef896 6.10.14-linuxkit #1 SMP Fri Nov 29 17:22:03 UTC 2024 aarch64 Linux
Linux 4013555a9d85 6.10.14-linuxkit #1 SMP Fri Nov 29 17:22:03 UTC 2024 aarch64 GNU/Linux

Subsystem

No response

What steps will reproduce the bug?

I noticed strange discrepancy between the memory limits set for node.js containers and the actual values I get for heap_size_limit and total_available_size. I suspected these are the values used for V8 memory management, but I might be wrong.

I would like to get clarity on which values to observe, what to expect, and how to set safe limits so that garbage collection surely works. Most likely the problem is still on our side - just wanted to clear my doubts.

Steps to reproduce:

1. Start a Node.js container with docker run -it -m 256m node:26-bookworm /bin/sh
2. Within the container, check the memory given for the container cat /sys/fs/cgroup/memory.max -> 268435456 (=256 * 1024 * 1024)
3. Start node.js process, examine the heap size node -e "console.log(v8.getHeapStatistics().heap_size_limit)" -> 274726912 (=262 * 1024 * 1024)

How often does it reproduce? Is there a required condition?

Always (the OOM kills and out of memory errors vary, but we get them on daily basis).

What is the expected behavior? Why is that the expected behavior?

The heap size limit should always be smaller than the limit set for the container and reflect the total memory available for V8.

What do you see instead?

The heap size limit is reported higher than the memory limit set for the container.

Additional information

It seems heap_size_limit and total_available_size are computed by some kind of step function, with 274726912 maybe being some kind of lower bound (see below). I am unsure what are the exact memory boundaries V8 would get; and if this stepwise behavior can explain our OOM kills and/or memory allocation failures (V8 killed in garbage collecting phase). We generally would like to keep our Node.js container nimble and well below gigabyte, as we do not need much (e.g. we have containers where we never expect >128Mb being allocated and can still get OOM kills with 256 or larger limits).

The following steps show the behavior with altering memory limits:

for mem in 96 128 256 384 449 512 768 1024 1286 1536; do
    docker run --rm -it -m "${mem}m" node:26-bookworm \
        /usr/local/bin/node -e 'console.log(v8.getHeapStatistics().heap_size_limit)'
done

274726912
274726912
274726912
281018368
281018368
281018368
427819008
562036736
724566016
855638016

for mem in 96 128 256 384 449 512 768 1024 1286 1536; do
    docker run --rm -it -m "${mem}m" node:26-bookworm \
        /usr/local/bin/node -e 'console.log(v8.getHeapStatistics().total_available_size)'
done

272045872
272045848
272045872
278337312
278337328
278337328
425137968
559355696
721884968
852956976

I repeated the same with the latest Docker images for v26, v24, v22 and v20 branch, and similar results (not the exact same heap size limit but always higher than available) repeats for all of them. The problem suspectedly has always existed? For reference, cat /sys/fs/cgroup/memory.high reports as max. The same problem repeats in Kubernetes environment as well, but I used Docker setup here for the sake of easy repetition.

As for background context, these occasional out of memory errors have haunted us for almost two years already. Previously we suspected it was due to old libuv cgroups handling and later due pointer compression problems. Until recently the problem manifested itself as V8 killing itself due to heap allocation failures when doing garbage collection; and now more recently as OOM kills by Kubernetes. Now that no known issues with libuv cgroups or pointer compression exist, we wanted to see what the metrics reveal. We have tried the usual tricks of limiting old and new space sizes, but to our understanding, now that sizing by cgroups limits should work, this is no longer best practice.

Bắt đầu bằng cách tái hiện sự cố với lệnh Docker được cung cấp, sau đó so sánh /sys/fs/cgroup/memory.max với heap_size_limit và total_available_size của V8 trên tất cả các giới hạn bộ nhớ được liệt kê. Theo dõi cách Node.js và V8 xử lý cgroups v2 để xác định liệu các giới hạn được báo cáo là có chủ đích hay không an toàn. Công việc được xem là hoàn tất khi đã xác lập được mối quan hệ mong đợi và sự cố đã có bản sửa lỗi được xác nhận hoặc hướng dẫn được ghi chép.