Graylog2 / Graylog2/graylog2-server

Bound the node registry reads used by the cluster health checks

Open
#27,259 4 comments 0 reactions 0 assignees View on GitHub
bug Project triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Impact

Two checks in the System Overview health panel, **Certificates** and **OpenSearch Version** (both under Search Cluster), can be blocked on MongoDB reads that the health plugin cannot put a time limit on.

When one of those reads is slow or never returns, the check reports `unknown` instead of a definite state. An `unknown` check is dropped from the panel's rollup unless everything beside it is also unknown, so the section above it can stay green while a real problem goes unreported.

## Why the plugin cannot fix this itself

Each check gets a fixed time budget, and the framework can only ignore a late answer, not stop the work behind it. So every check has to impose its own limit on the call it makes. Most MongoDB-backed checks do that with the driver's `withTimeout`, which bounds the whole operation including retries. These two cannot, because the collection is a private field rather than something the caller can reach.

## What to do

Give each service a bounded read that a caller on a deadline can use. Either a timeout-carrying overload alongside the existing method, or expose a bounded read the callers can drive. Keep it additive so existing callers are untouched.

## Dependencies

None. This touches `AbstractNodeService` and `DataNodeMetadataServiceImpl` only, and conflicts with nothing currently in flight. It is deliberately split out of #26886, whose `ClusterAdapter` half is blocked on #26885 and touches the three search client classes instead.

## Consumers

Tracked on the Enterprise side in Graylog2/graylog-plugin-enterprise#15110, under "Bounded node registry reads".

Contributor guide

Open the contributing guide

Research direction

Start by reading AbstractNodeService and DataNodeMetadataServiceImpl, then compare nearby MongoDB-backed checks that use the driver's withTimeout. Add an additive bounded-read API for each service without changing existing callers; done means the Certificates and OpenSearch Version health checks can enforce their deadlines without unbounded registry reads.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mongodb
Domain
backend, databases, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.