Graylog2 / Graylog2/graylog2-server
Data Node metrics ISM policy lacks ism_template, leaving new backing indices unmanaged
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## Expected Behavior
Every new backing index created by rollover of the `gl-datanode-metrics` data stream should automatically be managed by the Graylog-created `gl-datanode-metrics-ism` policy.
The policy should contain an `ism_template` matching future `.ds-gl-datanode-metrics-*` backing indices, or Graylog should otherwise associate the policy with the data stream in a way that persists across rollovers.
The OpenSearch 2.19 documentation states that an ISM policy is applied to data-stream backing indices when they are created:
https://docs.opensearch.org/2.19/im-plugin/data-streams/#step-5-rollover-a-data-stream
## Current Behavior
Graylog creates the `gl-datanode-metrics-ism` policy with:
```json
{
"ism_template": null
}
```
An existing backing index can be manually managed, but the next backing index created by rollover is unmanaged.
In this instance:
* `.ds-gl-datanode-metrics-000272` was manually attached to the policy.
* A subsequent natural rollover created `.ds-gl-datanode-metrics-000273`.
* The cluster was green, with no unassigned shards or failed ISM actions.
* ISM explain for `.ds-gl-datanode-metrics-000273` returned:
```json
{
".ds-gl-datanode-metrics-000273": {
"index.plugins.index_state_management.policy_id": null,
"index.opendistro.index_state_management.policy_id": null,
"enabled": null
},
"total_managed_indices": 0
}
```
Manually attaching the policy succeeds:
```json
{
"updated_indices": 1,
"failures": false,
"failed_indices": []
}
```
Afterward, ISM explain reports the index as enabled and managed by `gl-datanode-metrics-ism`.
This demonstrates that the OpenSearch ISM engine can manage the backing index, but no ISM template exists to attach the policy automatically when the next backing index is created.
The Graylog 7.1.8 source appears to confirm the cause.
`ConfigureMetricsIndexSettings.createPolicy()` passes `null` as the final `ismTemplate` argument:
https://github.com/Graylog2/graylog2-server/blob/7.1.8/data-node/src/main/java/org/graylog/datanode/metrics/ConfigureMetricsIndexSettings.java#L136-L146
`Policy.IsmTemplate` documents that this property is required for automatically rolled-over data-stream indices to continue being managed:
https://github.com/Graylog2/graylog2-server/blob/7.1.8/graylog2-server/src/main/java/org/graylog2/indexer/datastream/policy/Policy.java#L49-L54
The same `null` constructor argument is currently still present on the `master` branch:
https://github.com/Graylog2/graylog2-server/blob/master/data-node/src/main/java/org/graylog/datanode/metrics/ConfigureMetricsIndexSettings.java
## Possible Solution
Create `gl-datanode-metrics-ism` with a non-null `Policy.IsmTemplate` whose index pattern covers future `.ds-gl-datanode-metrics-*` backing indices.
Please also add an integration test that:
1. Creates the metrics data stream and ISM policy.
2. Verifies that the policy contains the required automatic-assignment configuration.
3. Performs a data-stream rollover.
4. Verifies that the newly created backing index is automatically managed by `gl-datanode-metrics-ism`.
## Steps to Reproduce
1. Run Graylog Server and Graylog Data Node 7.1.8.
2. Verify that the `gl-datanode-metrics` data stream exists.
3. Query:
```text
GET /_plugins/_ism/policies/gl-datanode-metrics-ism
```
4. Observe that `.policy.ism_template` is `null`.
5. Attach the policy to the current backing index if necessary.
6. Wait for the normal rollover or perform one manually:
```text
POST /gl-datanode-metrics/_rollover
```
7. Query ISM explain for the newly created backing index:
```text
GET /_plugins/_ism/explain/.ds-gl-datanode-metrics-
```
8. Observe that its policy ID and `enabled` fields are `null`.
9. Manually attach `gl-datanode-metrics-ism` to the new index.
10. Observe that the manual attachment succeeds and ISM explain then reports the index as managed.
## Context
Unmanaged metrics backing indices do not execute Graylog's configured rollup and deletion lifecycle.
Every rollover therefore requires manual intervention. Without detection and manual attachment, backing indices and shards accumulate and can eventually trigger `cluster.max_shards_per_node` warnings or prevent the creation of additional indices.
## Your Environment
* Graylog Version: 7.1.8-1
* Graylog Data Node Version: 7.1.8-1
* Data Node cluster size: 3
* Cluster health during reproduction: green
* Unassigned shards during reproduction: 0
* Java version: Eclipse Temurin OpenJDK 21.0.12+8 LTS
* Java source: JDK bundled with Graylog Data Node
* Java binary: `/usr/share/graylog-datanode/jvm/bin/java`
* OpenSearch Version: 2.19.5
* MongoDB Version: 8.2.12
* Operating System: Ubuntu 24.04.4 LTS
* Browser version: N/A
## Checklist
* [x] This issue fix needs to be backported.
* [ ] Does this issue have **security** implications?
Contributor guide
Assessment
This issue has not been assessed yet.