Graylog2 / Graylog2/graylog2-server
Health Check: Extend the bounded, cancellable read pattern to the remaining ClusterAdapter methods
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## Impact
Several checks in the System Overview health panel read the search backend without any time limit on the call: **Storage**, **CPU**, **Memory** and **Quorum** under Search Cluster, plus **Shard count** under Index Management.
When the search backend is unreachable, frozen, or just slow, those checks report `unknown` instead of a definite state. An `unknown` check is dropped from the panel's rollup unless everything beside it is also unknown, so the section above it can stay green while the backend is in trouble.
Graylog2/graylog2-server#26885 fixed the one check that caught it, **State**, by adding a bounded `Cluster.health(java.time.Duration)`. The others still have no way to bound their reads.
## Why a config setting does not cover it
`health()` cannot be bounded by configuration: `elasticsearch_connect_timeout` (10s) and `elasticsearch_socket_timeout` (60s) are applied **per host**, and the REST client works through every configured node inside a single call, so the worst case is a multiple of both. The same is true of every other `ClusterAdapter` read.
This section is the canonical statement of the problem. The consumer issue lives in a private repository, so the rationale is kept here.
## Dependencies and order
**Depends on #26885** and should land after it. Both touch the same three client classes (`ClusterAdapterOS2`, `ClusterAdapterES7`, `ClusterAdapterOS`), so starting first guarantees conflicts and risks diverging from the shape that PR settles on.
The equivalent work for the MongoDB-backed node registry is split out into #27259. It depends on nothing and can proceed in parallel.
## What to do
### `ClusterAdapter` reads
Extend the bounded, cancellable pattern to the reads that health checks actually depend on. Facade names are on `Cluster`, adapter names on `ClusterAdapter`:
- `getDiskUsageStats()` / `diskUsageStats()`, read by `SearchClusterStorageHealthReporter`
- `getNodesUtilization()` / `nodesUtilization()`, read by `SearchClusterCpuHealthReporter` and `SearchClusterMemoryHealthReporter` through a shared `SearchClusterNodeUtilizationCache`, so this is one call site rather than two. (This accessor was renamed from `getNodesStats()` by Graylog2/graylog2-server#26625, which is merged.)
- `countOfClusterManagerEligibleNodes()`, read by `SearchClusterQuorumHealthReporter`
- `clusterShardAllocation()` and `nodesInfo()`, both read by `ShardCountHealthReporter`, which arrives with Graylog2/graylog-plugin-enterprise#14917
`clusterHealthStats()` is deliberately **not** in scope. No health reporter reads it. Its callers are `OpensearchProcessImpl`, `ShardsMetricsSupplier` and `IndexerClusterResource`, none of which runs on the health budget. Add it here if that changes.
Keep the overloads additive so existing callers are untouched, as #26885 did.
## Consumers
Tracked on the Enterprise side in Graylog2/graylog-plugin-enterprise#15110, under "Bounded `ClusterAdapter` reads".
Contributor guide
Research direction
Start after #26885 lands, then inspect ClusterAdapterOS2, ClusterAdapterES7, and ClusterAdapterOS alongside the bounded Cluster.health(java.time.Duration) pattern. Trace the listed health reporters and ShardCountHealthReporter to confirm the adapter reads they use. Done means additive bounded, cancellable overloads cover disk usage, node utilization, eligible-node count, shard allocation, and node info without adding clusterHealthStats().
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100