Graylog2 / Graylog2/graylog2-server

NullPointerException – Error while expiring GELF chunk entries

Open
#9,061 1 comment 2 reactions 0 assignees View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

```
2020-09-30 10:27:34,759 WARN : org.graylog2.inputs.codecs.GelfChunkAggregator - Error while expiring GELF chunk entries
java.lang.NullPointerException: null
at java.util.concurrent.ConcurrentSkipListMap.remove(ConcurrentSkipListMap.java:1991) ~[?:1.8.0_212]
at java.util.concurrent.ConcurrentSkipListSet.remove(ConcurrentSkipListSet.java:259) ~[?:1.8.0_212]
at org.graylog2.inputs.codecs.GelfChunkAggregator.getAndCleanupEntry(GelfChunkAggregator.java:206) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfChunkAggregator.expireEntry(GelfChunkAggregator.java:195) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfChunkAggregator.access$200(GelfChunkAggregator.java:49) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfChunkAggregator$ChunkEvictionTask.run(GelfChunkAggregator.java:296) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_212]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_212]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
```

## Expected Behavior

Expired chunk entries should be expired without error.

## Current Behavior

Once the bug is triggered, the error above is logged each second and chunks are not being evicted.

## Possible Solution

It looks to me that `GelfChunkAggregator.getAndCleanupEntry` is modifying two data structures (`chunks` and `sortedEvictionSet`). It is being called from two threads (at least) via `checkForCompletion()` and the `ChunkEvictionTask`. `getAndCleanupEntry` does not look properly locked/synchronised to me (but then, I don't know much Java).

## Steps to Reproduce (for bugs)

I could not find a specific cause, yet. But looks like a threading issue, see above.

## Context

There are like 10k msgs/s coming in through UDP. So due to the nature of UDP chunks are being lost probably rather regularly.

## Your Environment

* Graylog Version: Graylog 3.0.2+1686930 (yeah, I know it's old. But the specific code hasn't changed in like 5 years)
* Java Version: Oracle Corporation 1.8.0_212
* Elasticsearch Version: 5.6.16
* MongoDB Version: 3.4
* Operating System: Linux 5.4.55
* Browser version:

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.