Graylog2 / Graylog2/graylog2-server
Cluster Configuration: Data Node metrics fetch fails with HTTP 500 on okhttp TCP connect timeout (InterruptedIOException) surfaced via AnyExceptionClassMapper
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## Summary
On System → Cluster Configuration, loading Data Node metrics intermittently fails with a UI popup:
> Could not load Data Node metrics
> Loading metrics for Data Node "" failed: FetchError: There was an error fetching a resource: Internal Server Error. Additional information: timeout.
Server-side, the request to the Data Node REST proxy times out in the **TCP connect phase** and the resulting `java.io.InterruptedIOException: timeout` is not mapped, so `AnyExceptionClassMapper` returns a bare HTTP 500 (message "timeout"), which the frontend renders as the popup above.
This has been present since **7.1.1** and still reproduces on **7.1.7** (latest).
## Environment
- Graylog: 7.1.7 (regression first observed on 7.1.1)
- Data Node / OpenSearch: 7.1.7
- MongoDB: 7.0.30
- OS: Ubuntu 24.04.4 LTS
- Topology: separate Graylog server and Data Node host(s)
## Steps to reproduce
1. Open System → Cluster Configuration.
2. Observe the "Could not load Data Node metrics … Additional information: timeout" popup for a Data Node.
3. Check `/var/log/graylog-server/server.log` — an `AnyExceptionClassMapper` error is logged at the same time.
Reproduces on a fully settled server (no restart in progress), triggered directly by opening the page.
## Expected
A Data Node metrics-fetch timeout should be handled gracefully (e.g. a "metrics temporarily unavailable" state / appropriate status code), not returned as an unmapped HTTP 500.
## Actual
HTTP 500 with body message "timeout"; UI shows a generic fetch error.
## Server log / stack trace (7.1.7, settled)
```
ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.io.InterruptedIOException: timeout
at okhttp3.internal.connection.RealCall.timeoutExit(RealCall.kt:461)
at okhttp3.internal.connection.RealCall.callDone(RealCall.kt:423)
at okhttp3.internal.connection.RealCall.noMoreExchanges$okhttp(RealCall.kt:381)
at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:234)
at okhttp3.internal.connection.RealCall.execute(RealCall.kt:178)
at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.request(DatanodeRestApiProxy.java:150)
at org.graylog2.rest.resources.datanodes.DataNodeRestApiProxyResource.request(DataNodeRestApiProxyResource.java:125)
at org.graylog2.rest.resources.datanodes.DataNodeRestApiProxyResource.requestPost(DataNodeRestApiProxyResource.java:96)
... (jersey/grizzly dispatch) ...
Caused by (connect phase):
at okhttp3.internal.connection.ConnectPlan.connectSocket(ConnectPlan.kt:276)
at okhttp3.internal.connection.ConnectPlan.connectTcp(ConnectPlan.kt:135)
at okhttp3.internal.connection.FastFallbackExchangeFinder$launchTcpConnect$1.runOnce(FastFallbackExchangeFinder.kt:141)
at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:148)
at java.base/java.net.Socket.connect(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl.connect(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl.endConnect(Unknown Source)
```
## Analysis
- The timeout bottoms out in `ConnectPlan.connectSocket` / `NioSocketImpl.endConnect` via okhttp's `FastFallbackExchangeFinder` (Happy Eyeballs) — i.e. the okhttp client in `DatanodeRestApiProxy` cannot establish a TCP socket to the Data Node within the connect timeout. It is **not** a read/response-side stall (no `Source.read` / `readResponseHeaders` frames).
- Because the exception is unmapped, `AnyExceptionClassMapper` returns HTTP 500, which the UI displays as a generic fetch error rather than a meaningful "Data Node unreachable / metrics unavailable" message.
## Two issues, arguably
1. The connect timeout to the Data Node REST endpoint itself (root cause — may be aggravated by a short connect timeout on the `DatanodeRestApiProxy` okhttp client and/or the Happy-Eyeballs connect path).
2. Error handling: a Data Node proxy timeout should not surface as an unmapped 500; it should be caught and returned as a graceful, typed error/status.
Contributor guide
Research direction
Start with DatanodeRestApiProxy.java and DataNodeRestApiProxyResource.java, tracing the request path shown in the stack trace and how AnyExceptionClassMapper handles the InterruptedIOException. Reproduce the metrics request from System → Cluster Configuration while checking server.log. Done means the timeout has a deliberate, user-visible handling path rather than an unmapped HTTP 500, with the intended connect-timeout behavior clarified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100