netdata / netdata/netdata

[Bug]: system.ram counts static hugepages as "used" memory, causing false alerts on KVM hypervisors with most of the RAM in HPs

Open
#22,867 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs triage
Dominant language
Go
Stars
80.6k
Forks
6.6k
Avg merge
15h 12m
Merged PRs (30d)
305

Description

Bug description

On KVM/QEMU hypervisors with pre-allocated hugepages, system.ram reports hugepages as "used" memory. This makes the ram_in_use health alert effectively useless, as it always shows near-critical utilization even when the system is under no memory pressure.

This is analogous to the ZFS ARC issue that was fixed in #12843 / #12847 -- specialized memory that is pre-allocated but not indicative of real memory pressure is counted as "used."

Example from a 1 TB hypervisor running RHEL 9.6 EUS with Netdata v2.10.3:

grep -E 'MemTotal|MemFree|MemAvailable|Huge' /proc/meminfo

MemTotal: 1055902584 kB
MemFree: 23681988 kB
MemAvailable: 37933000 kB
HugePages_Total: 458752
HugePages_Free: 384000
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 939524096 kB
Total RAM: ~1 TB

Hugepages reserved: 896 GB (85% of total), pre-allocated for KVM guests
Hugepages actually in use by VMs: ~146 GB (only 74,752 of 458,752 pages occupied)
Hugepages idle: ~750 GB
MemAvailable: ~37 GB
Netdata system.ram reports "used": ~95%+

The ram_in_use alert formula $used * 100 / ($used + $cached + $free + $buffers) does not account for hugepages, so it fires at the default 90% threshold even though the system has hundreds of GB of idle hugepages.

Expected behavior

system.ram should expose hugepages as a separate dimension (similar to how ZFS ARC was handled in #12847), so that:

The chart visually distinguishes hugepage memory from genuinely used memory.
The ram_in_use alert can exclude hugepages from its calculation, or a used_ram_to_ignore variable can be set to subtract Hugetlb from /proc/meminfo.

Steps to reproduce

Installa netdata from rpm (official repos) and check the RAM usage

Installation method

manual setup of official DEB/RPM packages

System info
[raistlin@daltigoth ~]$ uname -a; grep -HvE "^#|URL" /etc/*release
Linux daltigoth 5.14.0-570.123.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jun 19 03:08:16 EDT 2026 x86_64 x86_64 x86_64 GNU/Linux
/etc/os-release:NAME="Red Hat Enterprise Linux"
/etc/os-release:VERSION="9.6 (Plow)"
/etc/os-release:ID="rhel"
/etc/os-release:ID_LIKE="fedora"
/etc/os-release:VERSION_ID="9.6"
/etc/os-release:PLATFORM_ID="platform:el9"
/etc/os-release:PRETTY_NAME="Red Hat Enterprise Linux 9.6 (Plow)"
/etc/os-release:ANSI_COLOR="0;31"
/etc/os-release:LOGO="fedora-logo-icon"
/etc/os-release:CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
/etc/os-release:
/etc/os-release:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
/etc/os-release:REDHAT_BUGZILLA_PRODUCT_VERSION=9.6
/etc/os-release:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
/etc/os-release:REDHAT_SUPPORT_PRODUCT_VERSION="9.6"
/etc/redhat-release:Red Hat Enterprise Linux release 9.6 (Plow)
/etc/system-release:Red Hat Enterprise Linux release 9.6 (Plow)
Netdata build info
/etc/system-release:Red Hat Enterprise Linux release 9.6 (Plow)
[raistlin@daltigoth ~]$ netdata -W buildinfo
time=2026-06-25T11:05:26.020+02:00 comm=netdata source=daemon level=info errno="13, Permission denied" tid=3248826  msg="CONFIG: cannot open file '/etc/netdata/stream.conf'. Using internal defaults."
time=2026-06-25T11:05:26.020+02:00 comm=netdata source=daemon level=notice tid=3248826  msg="CONFIG: cannot load user config '/etc/netdata/stream.conf'. Will try stock config."
time=2026-06-25T11:05:26.021+02:00 comm=netdata source=collector level=error errno="13, Permission denied" tid=3248826  msg="Runtime directory '/tmp/netdata' is not writable, falling back to '/tmp'"
time=2026-06-25T11:05:26.023+02:00 comm=netdata source=daemon level=error errno="13, Permission denied" tid=3248826  msg="SYSTEM INFO: System info script /usr/libexec/netdata/plugins.d/system-info.sh not found or not readable."
Packaging:
    Netdata Version ____________________________________________ : v2.10.3
    Installation Type __________________________________________ : binpkg-rpm
    Package Architecture _______________________________________ : x86_64
    Package Distro _____________________________________________ :  
    Configure Options __________________________________________ : cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_STANDARD=11 -DCMAKE_CXX_STANDARD=17 -DBUILD_SHARED_LIBS=OFF -DCMAKE_C_FLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fexceptions -ffunction-sections -fdata-sections -Wno-builtin-macro-redefined -fno-omit-frame-pointer -funwind-tables -fasynchronous-unwind-tables' -DCMAKE_CXX_FLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fexceptions -ffunction-sections -fdata-sections -Wno-builtin-macro-redefined -fno-omit-frame-pointer -funwind-tables -fasynchronous-unwind-tables' -DCMAKE_COMPILE_DEFINITIONS='_GNU_SOURCE' -DCMAKE_EXE_LINKER_FLAGS='-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -fexceptions -ffunction-sections -fdata-sections -Wno-builtin-macro-redefined -rdynamic' -DCMAKE_SHARED_LINKER_FLAGS='-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1'
Default Directories:
    User Configurations ________________________________________ : /etc/netdata
    Stock Configurations _______________________________________ : /usr/lib/netdata/conf.d
    Ephemeral Databases (metrics data, metadata) _______________ : /var/cache/netdata
    Permanent Databases ________________________________________ : /var/lib/netdata
    Plugins ____________________________________________________ : /usr/libexec/netdata/plugins.d
    Static Web Files ___________________________________________ : /usr/share/netdata/web
    Log Files __________________________________________________ : /var/log/netdata
    Lock Files _________________________________________________ : /var/lib/netdata/lock
    Home _______________________________________________________ : /var/lib/netdata
Operating System:
    Kernel _____________________________________________________ : 
    Kernel Version _____________________________________________ : 
    Operating System ___________________________________________ : 
    Operating System ID ________________________________________ : 
    Operating System ID Like ___________________________________ : 
    Operating System Version ___________________________________ : 
    Operating System Version ID ________________________________ : 
    Detection __________________________________________________ : 
Hardware:
    CPU Cores __________________________________________________ : 
    CPU Frequency ______________________________________________ : 
    RAM Bytes __________________________________________________ : 
    Disk Capacity ______________________________________________ : 
    CPU Architecture ___________________________________________ : 
    Virtualization Technology __________________________________ : 
    Virtualization Detection ___________________________________ : 
Container:
    Container __________________________________________________ : 
    Container Detection ________________________________________ : 
    Container Orchestrator _____________________________________ : none
    Container Operating System _________________________________ : 
    Container Operating System ID ______________________________ : 
    Container Operating System ID Like _________________________ : 
    Container Operating System Version _________________________ : 
    Container Operating System Version ID ______________________ : 
    Container Operating System Detection _______________________ : 
Features:
    Built For __________________________________________________ : Linux
    Netdata Cloud ______________________________________________ : YES
    Health (trigger alerts and send notifications) _____________ : YES
    Streaming (stream metrics to parent Netdata servers) _______ : YES
    Back-filling (of higher database tiers) ____________________ : YES
    Replication (fill the gaps of parent Netdata servers) ______ : YES
    Streaming and Replication Compression ______________________ : YES (zstd lz4 gzip)
    Contexts (index all active and archived metrics) ___________ : YES
    Tiering (multiple dbs with different metrics resolution) ___ : YES (5)
    Machine Learning ___________________________________________ : YES
    Memory Allocator ___________________________________________ : system
Database Engines:
    dbengine (compression) _____________________________________ : YES (zstd lz4)
    alloc ______________________________________________________ : YES
    ram ________________________________________________________ : YES
    none _______________________________________________________ : YES
Connectivity Capabilities:
    ACLK (Agent-Cloud Link: MQTT over WebSockets over TLS) _____ : YES
    static (Netdata internal web server) _______________________ : YES
    WebRTC (experimental) ______________________________________ : NO
    Native HTTPS (TLS Support) _________________________________ : YES
    TLS Host Verification ______________________________________ : YES
Libraries:
    LZ4 (extremely fast lossless compression algorithm) ________ : YES
    ZSTD (fast, lossless compression algorithm) ________________ : YES
    zlib (lossless data-compression library) ___________________ : YES
    Brotli (generic-purpose lossless compression algorithm) ____ : NO
    protobuf (platform-neutral data serialization protocol) ____ : YES (system)
    OpenSSL (cryptography) _____________________________________ : YES
    libdatachannel (stand-alone WebRTC data channels) __________ : NO
    JSON-C (lightweight JSON manipulation) _____________________ : YES
    libcap (Linux capabilities system operations) ______________ : NO
    libcrypto (cryptographic functions) ________________________ : YES
    libyaml (library for parsing and emitting YAML) ____________ : YES
    libmnl (library for working with netfilter) ________________ : YES
    stacktraces (library for getting stack traces) _____________ : libbacktrace (mmap, threads, data)
Plugins:
    apps (monitor processes) ___________________________________ : YES
    cgroups (monitor containers and VMs) _______________________ : YES
    cgroup-network (associate interfaces to CGROUPS) ___________ : YES
    proc (monitor Linux systems) _______________________________ : YES
    tc (monitor Linux network QoS) _____________________________ : YES
    diskspace (monitor Linux mount points) _____________________ : YES
    freebsd (monitor FreeBSD systems) __________________________ : NO
    macos (monitor MacOS systems) ______________________________ : NO
    windows (monitor Windows systems) __________________________ : NO
    statsd (collect custom application metrics) ________________ : YES
    timex (check system clock synchronization) _________________ : YES
    idlejitter (check system latency and jitter) _______________ : YES
    bash (support shell data collection jobs - charts.d) _______ : YES
    debugfs (kernel debugging metrics) _________________________ : YES
    cups (monitor printers and print jobs) _____________________ : YES
    ebpf (monitor system calls) ________________________________ : YES
    freeipmi (monitor enterprise server H/W) ___________________ : YES
    network-viewer (monitor TCP/UDP IPv4/6 sockets) ____________ : YES
    systemd-journal (monitor journal logs) _____________________ : YES
    windows-events (monitor Windows events) ____________________ : NO
    nfacct (gather netfilter accounting) _______________________ : NO
    perf (collect kernel performance events) ___________________ : YES
    slabinfo (monitor kernel object caching) ___________________ : YES
    Xen ________________________________________________________ : NO
    Xen VBD Error Tracking _____________________________________ : NO
Exporters:
    AWS Kinesis ________________________________________________ : NO
    GCP PubSub _________________________________________________ : NO
    MongoDB ____________________________________________________ : YES
    Prometheus (OpenMetrics) Exporter __________________________ : YES
    Prometheus Remote Write ____________________________________ : YES
    Graphite ___________________________________________________ : YES
    Graphite HTTP / HTTPS ______________________________________ : YES
    JSON _______________________________________________________ : YES
    JSON HTTP / HTTPS __________________________________________ : YES
    OpenTSDB ___________________________________________________ : YES
    OpenTSDB HTTP / HTTPS ______________________________________ : YES
    All Metrics API ____________________________________________ : YES
    Shell (use metrics in shell scripts) _______________________ : YES
Debug/Developer Features:
    Trace All Netdata Allocations (with charts) ________________ : NO
    Developer Mode (more runtime checks, slower) _______________ : NO
Runtime Information:
    Profile ____________________________________________________ : standalone
    Stream Parent (accept data from Children) __________________ : NO
    Stream Child (send data to a Parent) _______________________ : NO
    Total System Memory ________________________________________ : 1081244246016
    Available System Memory ____________________________________ : 39119650816
[raistlin@daltigoth ~]$
Additional info

Also, it is important to remember that there are various types of hugepages:

  • hugepages can be static or transparent (anonhugepages)
  • hugepages can be either 2MB or 1GB in size and a Linux system can have 4k pages, 2MB HPs and 1GB HPs at the same time and on the same system.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the system.ram entry point and the ram_in_use alert, using the /proc/meminfo Hugetlb value described in the report. Compare the handling of specialized memory with the ZFS ARC changes in #12843 and #12847. Done means hugepages are visibly distinct from genuinely used memory and the alert no longer counts idle hugepages as pressure.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux
Domain
observability, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.