Graylog2 / Graylog2/graylog2-server

Clean-up Data Node metrics datastream: wrong mapping type, inconsistent unit, and misnamed field

Open
#26,783 0 comments 0 reactions 0 assignees View on GitHub
backend bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

Three long-standing minor problems in `NodeStatMetrics` affect what gets written into the `gl-datanode-metrics` datastream.

All three were found while reviewing #26732. None of them affect the Cluster Configuration page. Only the first is really impactful; but may as well fix the other 2 alongside.

## 1. `mem_heap_used_bytes` and `mem_heap_max` are truncated to whole GiB

`data-node/src/main/java/org/graylog/datanode/metrics/NodeStatMetrics.java`

```java
MEM_HEAP_USED_BYTES("integer", new RollupAction.IsmRollup.AvgMetric(), "$.jvm.mem.heap_used_in_bytes", NodeStatMetrics::bytesToGb),
MEM_HEAP_MAX("integer", new RollupAction.IsmRollup.AvgMetric(), "$.jvm.mem.heap_max_in_bytes", NodeStatMetrics::bytesToGb),
```

Both declare mapping type `integer` but are converted with `bytesToGb`, which returns a `float`. The mapping type is what builds the datastream mapping in `ConfigureMetricsIndexSettings.createMappings()`, and OpenSearch `integer` fields coerce by truncating toward zero.

Result: a 512 MiB heap is stored as `0`, and a 1.5 GiB heap is stored as `1`. Any heap below 1 GiB records as zero, and everything else loses all sub-GiB resolution. The ISM rollup in `ConfigureMetricsIndexSettings.ismRollupState()` then averages the truncated values, so the daily rollup index inherits the error.

`MEM_TOTAL_USED_BYTES` in the same enum is declared `float` and is correct, so this looks like an oversight rather than intent.

Suggested fix: change both to `float`.

Note that changing the mapping only affects newly created backing indices. The template is re-applied on manager node startup, but OpenSearch will not retype fields in existing backing indices, so the datastream needs a rollover before the corrected mapping takes effect.

## 2. `mem_free` is in MiB while every other memory field is in GiB

```java
MEM_FREE("float", new RollupAction.IsmRollup.AvgMetric(), "$.os.mem.free_in_bytes", NodeStatMetrics::bytesToMb),
MEM_TOTAL("float", new RollupAction.IsmRollup.AvgMetric(), "$.os.mem.total_in_bytes", NodeStatMetrics::bytesToGb),
```

`MEM_FREE` is the only memory metric using `bytesToMb`. Every other one uses `bytesToGb`. That makes `mem_free` and `mem_total` incomparable, so an expression like `mem_total - mem_free` silently produces a meaningless number.

Suggested fix: use `bytesToGb` for consistency.

## 3. `disk_used` actually holds total disk, not used disk

```java
DISK_USED("float", new RollupAction.IsmRollup.AvgMetric(), "$.fs.total.total_in_bytes", NodeStatMetrics::bytesToGb),
```

The field named `disk_used` reads `$.fs.total.total_in_bytes`, which is total disk size. There is no field holding actual used disk.

The Cluster Configuration page is not affected because the frontend derives used space itself from `total - available` in `calculateUsedFsBytes`. Only the indexed field and anything querying it directly are wrong.

Suggested fix: rename the field to `disk_total`, or keep the name and change it to report genuinely used bytes. Either is a breaking change for anyone with saved searches or custom dashboards over `gl-datanode-metrics`, so the choice is worth a deliberate decision rather than a silent change.

## Scope

None of `mem_heap_used_bytes`, `mem_heap_max`, `mem_free`, or `disk_used` are referenced by the bundled sample dashboards under `graylog2-web-interface/src/components/datanode/ClusterManagement/sample-dashboards/`, which only use `disk_free`, `cpu_load`, `mem_heap_used`, and the thread pool fields. So the impact is limited to the contents of the datastream and to user built searches over it.

Contributor guide

Open the contributing guide

Research direction

Start in data-node/src/main/java/org/graylog/datanode/metrics/NodeStatMetrics.java and trace how ConfigureMetricsIndexSettings.createMappings() and ismRollupState() consume these definitions. Compare the memory conversions and field names with the datastream mapping, then inspect the bundled sample dashboards to confirm their stated scope. Done means the mapping and units are consistent and the disk field’s breaking-change choice is explicitly resolved, with rollover implications understood.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.