Graylog2 / Graylog2/graylog2-server

POST requests with h2c header result in timeout

Open
#24,534 1 comment 0 reactions 0 assignees View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

A `POST` request to any given Graylog API endpoint results in a timeout, with a 500 response code.

## Expected Behavior

No timeouts, or 500 internal server errors. Like when the `Upgrade: h2c` header is not present:

```sh
curl -v -X POST -H "Content-type:application/json" -d '{
"queries": [
{
"query": {
"type": "elasticsearch",
"query_string": "resource:/login"
},
"timerange": {
"type": "relative",
"range": 36000
},
"search_types": [
{
"id": "messages",
"type": "messages",
"limit": 5,
"offset": 0
}
]
}
]
}' -H "X-Requested-By: me" "http://admin:admin@localhost:9000/api/views/search/sync"

Note: Unnecessary use of -X or --request, POST is already inferred.
* Host localhost:9000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
* Trying [::1]:9000...
* connect to ::1 port 9000 from ::1 port 56646 failed: Connection refused
* Trying 127.0.0.1:9000...
* Connected to localhost (127.0.0.1) port 9000
* Server auth using Basic with user 'admin'
> POST /api/views/search/sync HTTP/1.1
> Host: localhost:9000
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/8.7.1
> Accept: */*
> Content-type:application/json
> X-Requested-By: me
> Content-Length: 222
>
* upload completely sent off: 222 bytes
< HTTP/1.1 200 OK
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Graylog-Node-ID: 10b4a276-d245-49de-a581-f21e391b2ed4
< Content-Type: application/json
< Content-Length: 1519
<
* Connection #0 to host localhost left intact
{"execution":{"done":true,"cancelled":false,"completed_exceptionally":true},"results":{"80207766-98fc-4491-8471-1f58fb5e3015":{"query":{"id":"80207766-98fc-4491-8471-1f58fb5e3015","timerange":{"range":36000,"type":"relative"},"filter":{"type":"or","filters":[{"type":"stream","id":"68ff416a36de981339f9b5eb"},{"type":"stream","id":"000000000000000000000001"},{"type":"stream","id":"68fb2073c0ef687a32a5ea16"}]},"filters":[],"query":{"type":"elasticsearch","query_string":"resource:/login"},"search_types":[{"timerange":null,"query":null,"streams":[],"stream_categories":[],"type":"messages","id":"messages","name":null,"limit":5,"offset":0,"sort":null,"fields":[],"decorators":[],"filter":null,"filters":[]}]},"execution_stats":{"duration":81,"timestamp":"2025-12-10T15:12:41.555Z","effective_timerange":{"from":"2025-12-10T05:12:41.443Z","to":"2025-12-10T15:12:41.443Z","type":"absolute"}},"search_types":{},"errors":[{"type":"search_type","description":"OpenSearch exception [type=search_phase_execution_exception, reason=all shards failed].","search_type_id":"messages","query_id":"80207766-98fc-4491-8471-1f58fb5e3015"}],"state":"COMPLETED"}},"errors":[{"type":"search_type","description":"OpenSearch exception [type=search_phase_execution_exception, reason=all shards failed].","search_type_id":"messages","query_id":"80207766-98fc-4491-8471-1f58fb5e3015"}],"progress":0,"id":"69398de9cb1e50edbbf40d1c","search_id":"69398de9cb1e50edbbf40d1a","owner":"admin","executing_node":"10b4a276-d245-49de-a581-f21e391b2ed4"}
```

## Current Behavior

_Note the `--http2` option added to the `curl` command._

```sh
curl --http2 -v -X POST -H "Content-type:application/json" -d '{
"queries": [
{
"query": {
"type": "elasticsearch",
"query_string": "resource:/login"
},
"timerange": {
"type": "relative",
"range": 36000
},
"search_types": [
{
"id": "messages",
"type": "messages",
"limit": 5,
"offset": 0
}
]
}
]
}' -H "X-Requested-By: me" "http://admin:admin@localhost:9000/api/views/search/sync"

Note: Unnecessary use of -X or --request, POST is already inferred.
* Host localhost:9000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
* Trying [::1]:9000...
* connect to ::1 port 9000 from ::1 port 56649 failed: Connection refused
* Trying 127.0.0.1:9000...
* Connected to localhost (127.0.0.1) port 9000
* Server auth using Basic with user 'admin'
> POST /api/views/search/sync HTTP/1.1
> Host: localhost:9000
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/8.7.1
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAAQAoAAAAAIAAAAA
> Content-type:application/json
> X-Requested-By: me
> Content-Length: 222
>
* upload completely sent off: 222 bytes
< HTTP/1.1 500 Internal Server Error
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Graylog-Node-ID: 10b4a276-d245-49de-a581-f21e391b2ed4
< Content-Length: 69
<
* Connection #0 to host localhost left intact
{"type":"ApiError","message":"java.util.concurrent.TimeoutException"}
```

## Possible Solution

Adding a Grizzly/Jersey filter to handle HTTP/2 upgrade requests (or perhaps upgrade dependencies? Not sure if it's possible, though)

## Steps to Reproduce (for bugs)

1. Send any `POST` request to any endpoint from the Graylog API that supports `POST` operations, and add the `Connection: Upgrade, HTTP2-Settings`, `Upgrade: h2c`, and `HTTP2-Settings` headers. The client can be `curl` like above, or you could also use this:

```java
# SimpleHttpClient.java
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.util.Base64;

public class SimpleHttpClient {

public static void main(String[] args) {
// Configuration
String url = "http://localhost:9000/api/views/search/sync";
String username = "admin";
String password = "admin";
String postData = """
{
"queries": [
{
"query": {
"type": "elasticsearch",
"query_string": "resource:/login"
},
"timerange": {
"type": "relative",
"range": 3600
},
"search_types": [
{
"id": "messages",
"type": "messages",
"limit": 5,
"offset": 0
}
]
}
]
}
""";

try {
// Create HTTP client
HttpClient client = HttpClient.newHttpClient();

// Encode credentials for basic auth
String auth = username + ":" + password;
String encodedAuth = Base64.getEncoder()
.encodeToString(auth.getBytes(StandardCharsets.UTF_8));

// Build the request
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.header("Authorization", "Basic " + encodedAuth)
.header("Content-Type", "application/json")
.header("X-Requested-By", "me")
.POST(HttpRequest.BodyPublishers.ofString(postData))
.build();

// Send the request and get response
HttpResponse response = client.send(
request,
HttpResponse.BodyHandlers.ofString()
);

// Print results
System.out.println("Response Code: " + response.statusCode());
System.out.println("\nResponse Body:");
System.out.println(response.body());

} catch (Exception e) {
System.err.println("Error occurred: " + e.getMessage());
e.printStackTrace();
}
}
}
```

and then

```sh
java SimpleHttpClient.java
```

2. Wait for a timeout.
## Related Issues

- https://github.com/eclipse-ee4j/jersey/issues/5812

- https://github.com/eclipse-ee4j/glassfish-grizzly/issues/2117

## Context

Development of a simple MCP client Java app that performs operations with the Graylog API, using the default `java.net.http.HttpClient`. MCP operations are almost always initiated by `POST` requests, and `HttpClient` defaults to HTTP/2.

## Your Environment

* Graylog Version: 7.0
* Java Version: 21
* OpenSearch Version: 2.19.3
* MongoDB Version: 7.0
* Operating System: macOS Tahoe 26.1
* Browser version: Firefox 146.0

## Checklist
[] This issue fix need to be backported.
[] Does this issue have **security** implications?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.