Graylog2 / Graylog2/graylog2-server

Cluster Configuration: Data Node metrics fetch fails with HTTP 500 on okhttp TCP connect timeout (InterruptedIOException) surfaced via AnyExceptionClassMapper

Open
#26,975 1 comment 0 reactions 0 assignees View on GitHub
bug frontend triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Summary
On System → Cluster Configuration, loading Data Node metrics intermittently fails with a UI popup:

> Could not load Data Node metrics
> Loading metrics for Data Node "" failed: FetchError: There was an error fetching a resource: Internal Server Error. Additional information: timeout.

Server-side, the request to the Data Node REST proxy times out in the **TCP connect phase** and the resulting `java.io.InterruptedIOException: timeout` is not mapped, so `AnyExceptionClassMapper` returns a bare HTTP 500 (message "timeout"), which the frontend renders as the popup above.

This has been present since **7.1.1** and still reproduces on **7.1.7** (latest).

## Environment
- Graylog: 7.1.7 (regression first observed on 7.1.1)
- Data Node / OpenSearch: 7.1.7
- MongoDB: 7.0.30
- OS: Ubuntu 24.04.4 LTS
- Topology: separate Graylog server and Data Node host(s)

## Steps to reproduce
1. Open System → Cluster Configuration.
2. Observe the "Could not load Data Node metrics … Additional information: timeout" popup for a Data Node.
3. Check `/var/log/graylog-server/server.log` — an `AnyExceptionClassMapper` error is logged at the same time.

Reproduces on a fully settled server (no restart in progress), triggered directly by opening the page.

## Expected
A Data Node metrics-fetch timeout should be handled gracefully (e.g. a "metrics temporarily unavailable" state / appropriate status code), not returned as an unmapped HTTP 500.

## Actual
HTTP 500 with body message "timeout"; UI shows a generic fetch error.

## Server log / stack trace (7.1.7, settled)

```
ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.io.InterruptedIOException: timeout
at okhttp3.internal.connection.RealCall.timeoutExit(RealCall.kt:461)
at okhttp3.internal.connection.RealCall.callDone(RealCall.kt:423)
at okhttp3.internal.connection.RealCall.noMoreExchanges$okhttp(RealCall.kt:381)
at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:234)
at okhttp3.internal.connection.RealCall.execute(RealCall.kt:178)
at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.request(DatanodeRestApiProxy.java:150)
at org.graylog2.rest.resources.datanodes.DataNodeRestApiProxyResource.request(DataNodeRestApiProxyResource.java:125)
at org.graylog2.rest.resources.datanodes.DataNodeRestApiProxyResource.requestPost(DataNodeRestApiProxyResource.java:96)
... (jersey/grizzly dispatch) ...
Caused by (connect phase):
at okhttp3.internal.connection.ConnectPlan.connectSocket(ConnectPlan.kt:276)
at okhttp3.internal.connection.ConnectPlan.connectTcp(ConnectPlan.kt:135)
at okhttp3.internal.connection.FastFallbackExchangeFinder$launchTcpConnect$1.runOnce(FastFallbackExchangeFinder.kt:141)
at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:148)
at java.base/java.net.Socket.connect(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl.connect(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl.endConnect(Unknown Source)
```

## Analysis
- The timeout bottoms out in `ConnectPlan.connectSocket` / `NioSocketImpl.endConnect` via okhttp's `FastFallbackExchangeFinder` (Happy Eyeballs) — i.e. the okhttp client in `DatanodeRestApiProxy` cannot establish a TCP socket to the Data Node within the connect timeout. It is **not** a read/response-side stall (no `Source.read` / `readResponseHeaders` frames).
- Because the exception is unmapped, `AnyExceptionClassMapper` returns HTTP 500, which the UI displays as a generic fetch error rather than a meaningful "Data Node unreachable / metrics unavailable" message.

## Two issues, arguably
1. The connect timeout to the Data Node REST endpoint itself (root cause — may be aggravated by a short connect timeout on the `DatanodeRestApiProxy` okhttp client and/or the Happy-Eyeballs connect path).
2. Error handling: a Data Node proxy timeout should not surface as an unmapped 500; it should be caught and returned as a graceful, typed error/status.

Contributor guide

Open the contributing guide

Research direction

Start with DatanodeRestApiProxy.java and DataNodeRestApiProxyResource.java, tracing the request path shown in the stack trace and how AnyExceptionClassMapper handles the InterruptedIOException. Reproduce the metrics request from System → Cluster Configuration while checking server.log. Done means the timeout has a deliberate, user-visible handling path rather than an unmapped HTTP 500, with the intended connect-timeout behavior clarified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.