Graylog2 / Graylog2/graylog2-server

[MISSING LOGS] Http keep-alive on HTTP GELF input with bulk-receive enabled causes lost messages

Open
#21,551 0 comments 0 reactions 0 assignees View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

When you try to send multiple log entries to Graylog using the HTTP GELF input using multiple HTTP requests. With bulk-receiving enabled on this input and http keep-alive enabled during the sending of the multiple HTTP requests, only the first one is processed and the rest is discarded.

## Expected Behavior
All the logs should be processed, not only the first one in the tcp connection.

## Current Behavior
Only the log entries from the first http request in the keep-alive tcp connection are processed.

## Possible Solution
Fix the internal HTTP handler in such a way, that when bulk-receiving is enabled, it properly uses the content-length header of the HTTP requests to determine if it should keep listening for more log lines or for more HTTP requests instead.

My guess is that something needs to be fixed in here:
https://github.com/Graylog2/graylog2-server/blob/master/graylog2-server/src/main/java/org/graylog2/inputs/transports/netty/LenientDelimiterBasedFrameDecoder.java

## Steps to Reproduce (for bugs)

1. Create a new HTTP GELF input
2. Enable bulk receiving on this input
3. Send 10 different logs using 10 different http requests with keepalive (so in the same connection)
4. Only 1 or 2 logs will be actually processed in graylog

Example script to reproduce bug:
```js
const GRAYLOG_URL = 'http://localhost:12201';

const randomid = Math.random().toString(36).substring(7);
console.log('Random ID:', randomid);

async function main() {
for (let i = 0; i < 10; i++) {
const message = {
version: '1.1',
timestamp: Date.now() / 1000,
host: 'wheremessage',
level: 6,
short_message: randomid + ' Short ' + i,
full_message: randomid + ' Long ' + i,
};

await fetch(GRAYLOG_URL + '/gelf', {
keepalive: true,
method: 'POST',
headers: {
'Connection': 'keep-alive',
'Content-Type': 'application/json',
'Accept-Encoding': 'gzip,deflate',
},
body: JSON.stringify(message),
}).then(response => console.log(response.status));
}
}

main();
```

## Context
I am currently using graylog to save all logs from both a frontend and backend. The frontend aggregates many logs into one http request, so I need the bulk receive option. But the backend uses many http requests for sending it's logs, however I cannot set 'Connection: Close' because graylog is running in AWS behind a load balancer. This load balancer forces the use of 'Connection: Keep-Alive'.

## Your Environment
Graylog is running in docker in an EC2 instance inside AWS behind a https load balancer that targets the http gelf port.

* Graylog Version: 6.1.6+a644883
* Java Version: Eclipse Adoptium 17.0.14 on Linux 6.1.119-129.201.amzn2023.aarch64
* OpenSearch Version: Not relevant
* MongoDB Version: Not relevant
* Operating System: AWS Linux ARM64
* Browser version: Not relevant

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.