Graylog2 / Graylog2/graylog2-server
NullPointerException – Error while expiring GELF chunk entries
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
```
2020-09-30 10:27:34,759 WARN : org.graylog2.inputs.codecs.GelfChunkAggregator - Error while expiring GELF chunk entries
java.lang.NullPointerException: null
at java.util.concurrent.ConcurrentSkipListMap.remove(ConcurrentSkipListMap.java:1991) ~[?:1.8.0_212]
at java.util.concurrent.ConcurrentSkipListSet.remove(ConcurrentSkipListSet.java:259) ~[?:1.8.0_212]
at org.graylog2.inputs.codecs.GelfChunkAggregator.getAndCleanupEntry(GelfChunkAggregator.java:206) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfChunkAggregator.expireEntry(GelfChunkAggregator.java:195) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfChunkAggregator.access$200(GelfChunkAggregator.java:49) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfChunkAggregator$ChunkEvictionTask.run(GelfChunkAggregator.java:296) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_212]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_212]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
```
## Expected Behavior
Expired chunk entries should be expired without error.
## Current Behavior
Once the bug is triggered, the error above is logged each second and chunks are not being evicted.
## Possible Solution
It looks to me that `GelfChunkAggregator.getAndCleanupEntry` is modifying two data structures (`chunks` and `sortedEvictionSet`). It is being called from two threads (at least) via `checkForCompletion()` and the `ChunkEvictionTask`. `getAndCleanupEntry` does not look properly locked/synchronised to me (but then, I don't know much Java).
## Steps to Reproduce (for bugs)
I could not find a specific cause, yet. But looks like a threading issue, see above.
## Context
There are like 10k msgs/s coming in through UDP. So due to the nature of UDP chunks are being lost probably rather regularly.
## Your Environment
* Graylog Version: Graylog 3.0.2+1686930 (yeah, I know it's old. But the specific code hasn't changed in like 5 years)
* Java Version: Oracle Corporation 1.8.0_212
* Elasticsearch Version: 5.6.16
* MongoDB Version: 3.4
* Operating System: Linux 5.4.55
* Browser version:
Contributor guide
Assessment
This issue has not been assessed yet.