Graylog2 / Graylog2/graylog2-server

Cluster Configuration page shows host-level CPU/memory instead of container-scoped values

Open
#27,264 6 comments 0 reactions 0 assignees View on GitHub
7.3 candidate bug test-day
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior
When Graylog server and data nodes run in Docker, the Cluster Configuration page's CPU and memory figures should reflect each container's own usage and limits.

## Current Behavior
The page shows host-level, and in one case flatly wrong, figures instead.

**CPU (both node types) — host-wide, not container-scoped:**
- Graylog server: `CpuLoadGauge` reads OSHI's `CentralProcessor.getSystemCpuLoadTicks()`, sourced from `/proc/stat`, a host-wide counter.
- Data node: `NodeStatMetrics` forwards OpenSearch's `os.cpu.percent` / `os.cpu.load_average.1m`, which OpenSearch's `OsProbe` also reads host-wide.

In a 2-node GL / 2-node DN cluster on one Docker host, all four nodes reported near-identical CPU — the host's, not each container's.

**Memory (data node) — host-wide, and inflated by disk cache:**
`NodeStatMetrics` also forwards OpenSearch's `os.mem.*` fields. `OsProbe` computes these from `com.sun.management.OperatingSystemMXBean.getFreePhysicalMemorySize()`/`getTotalPhysicalMemorySize()`, which on Linux maps to `/proc/meminfo`'s `MemFree`, not
`MemAvailable`. `MemFree` excludes reclaimable page cache/buffers, which Linux — and OpenSearch/Lucene in particular — fills aggressively with disk cache. A data node with plenty of free capacity showed 98%+ memory used, driven almost entirely by page
cache the OS would instantly reclaim if needed.

This produced two concrete false signals: a data node reported 15G total memory on a 16G host (host-wide, not its container's actual limit) and recommended raising the heap; separately, both data nodes showed 98%+ memory used with no real memory
pressure.

**Unaffected:** heap metrics (`jvm.memory.heap.*` on GL server, `jvm.mem.heap_*` on data node) are JVM-internal and correct.

## Possible Solution
OpenSearch's `OsProbe` already detects cgroups and exposes an `os.cgroup` block (`memory.limit_in_bytes`, `memory.usage_in_bytes`, `cpu.cfs_quota_micros`, `cpuacct.usage_nanos`) in `_nodes/stats` when running under a cgroup limit. `NodeStatMetrics`
currently ignores this block — wiring it in would fix the data node's CPU and memory figures, including the cache-inflation problem, since cgroup memory accounting excludes reclaimable cache differently than `MemFree` does. The Graylog server side
needs a separate cgroup-aware CPU read, since it has no equivalent field to fall back on.

## Steps to Reproduce
1. Run a 2-node Graylog server + 2-node data node cluster via Docker on one host.
2. Open Cluster Configuration.
3. Compare the reported CPU/memory to each container's actual `docker stats` or cgroup limits, and to `free -h`'s `available` column on the host.
4. Note that all nodes report near-identical, host-level CPU; data node memory shows 98%+ used despite ample free/reclaimable capacity.

## Context
Misleading resource figures led to an unnecessary heap-size recommendation for a data node, based on host memory rather than the container's actual limit, and a false impression of memory pressure driven by disk cache.
Related to https://github.com/Graylog2/quality-assurance/issues/82 - see also there for screenshots.

## Your Environment
* Graylog Version: 7.2.0-beta.1-1
* Java Version: 21
* OpenSearch Version: Data Node 7.2.0-beta-1.1
* MongoDB Version: 8.3
* Operating System: Docker on Linux
* Browser version: Firefox 154.0.1

## Checklist
- [ ] This issue fix need to be backported.
- [ ] Does this issue have **security** implications?

Contributor guide

Open the contributing guide

Research direction

Start with CpuLoadGauge and NodeStatMetrics, then trace the Cluster Configuration page's CPU and memory values. Inspect the OpenSearch _nodes/stats os.cgroup fields and OsProbe behavior on Linux containers. Done means server and data-node metrics reflect container scope and limits without changing the unaffected JVM heap metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java, linux
Domain
backend, infrastructure, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.