Graylog2 / Graylog2/graylog2-server
Cluster Configuration page shows host-level CPU/memory instead of container-scoped values
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## Expected Behavior
When Graylog server and data nodes run in Docker, the Cluster Configuration page's CPU and memory figures should reflect each container's own usage and limits.
## Current Behavior
The page shows host-level, and in one case flatly wrong, figures instead.
**CPU (both node types) — host-wide, not container-scoped:**
- Graylog server: `CpuLoadGauge` reads OSHI's `CentralProcessor.getSystemCpuLoadTicks()`, sourced from `/proc/stat`, a host-wide counter.
- Data node: `NodeStatMetrics` forwards OpenSearch's `os.cpu.percent` / `os.cpu.load_average.1m`, which OpenSearch's `OsProbe` also reads host-wide.
In a 2-node GL / 2-node DN cluster on one Docker host, all four nodes reported near-identical CPU — the host's, not each container's.
**Memory (data node) — host-wide, and inflated by disk cache:**
`NodeStatMetrics` also forwards OpenSearch's `os.mem.*` fields. `OsProbe` computes these from `com.sun.management.OperatingSystemMXBean.getFreePhysicalMemorySize()`/`getTotalPhysicalMemorySize()`, which on Linux maps to `/proc/meminfo`'s `MemFree`, not
`MemAvailable`. `MemFree` excludes reclaimable page cache/buffers, which Linux — and OpenSearch/Lucene in particular — fills aggressively with disk cache. A data node with plenty of free capacity showed 98%+ memory used, driven almost entirely by page
cache the OS would instantly reclaim if needed.
This produced two concrete false signals: a data node reported 15G total memory on a 16G host (host-wide, not its container's actual limit) and recommended raising the heap; separately, both data nodes showed 98%+ memory used with no real memory
pressure.
**Unaffected:** heap metrics (`jvm.memory.heap.*` on GL server, `jvm.mem.heap_*` on data node) are JVM-internal and correct.
## Possible Solution
OpenSearch's `OsProbe` already detects cgroups and exposes an `os.cgroup` block (`memory.limit_in_bytes`, `memory.usage_in_bytes`, `cpu.cfs_quota_micros`, `cpuacct.usage_nanos`) in `_nodes/stats` when running under a cgroup limit. `NodeStatMetrics`
currently ignores this block — wiring it in would fix the data node's CPU and memory figures, including the cache-inflation problem, since cgroup memory accounting excludes reclaimable cache differently than `MemFree` does. The Graylog server side
needs a separate cgroup-aware CPU read, since it has no equivalent field to fall back on.
## Steps to Reproduce
1. Run a 2-node Graylog server + 2-node data node cluster via Docker on one host.
2. Open Cluster Configuration.
3. Compare the reported CPU/memory to each container's actual `docker stats` or cgroup limits, and to `free -h`'s `available` column on the host.
4. Note that all nodes report near-identical, host-level CPU; data node memory shows 98%+ used despite ample free/reclaimable capacity.
## Context
Misleading resource figures led to an unnecessary heap-size recommendation for a data node, based on host memory rather than the container's actual limit, and a false impression of memory pressure driven by disk cache.
Related to https://github.com/Graylog2/quality-assurance/issues/82 - see also there for screenshots.
## Your Environment
* Graylog Version: 7.2.0-beta.1-1
* Java Version: 21
* OpenSearch Version: Data Node 7.2.0-beta-1.1
* MongoDB Version: 8.3
* Operating System: Docker on Linux
* Browser version: Firefox 154.0.1
## Checklist
- [ ] This issue fix need to be backported.
- [ ] Does this issue have **security** implications?
Contributor guide
Research direction
Start with CpuLoadGauge and NodeStatMetrics, then trace the Cluster Configuration page's CPU and memory values. Inspect the OpenSearch _nodes/stats os.cgroup fields and OsProbe behavior on Linux containers. Done means server and data-node metrics reflect container scope and limits without changing the unaffected JVM heap metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, java, linux
- Domain
- backend, infrastructure, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100