Graylog2 / Graylog2/graylog2-server

Cache rDNS results to improve throughput

Open
#11,977 9 comments 1 reaction 0 assignees View on GitHub
feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## What?
Cache the results of rDNS lookups to reduce resolver overhead and improve throughput.

## Why?
Currently Graylog calls `InetAddress.getCanonicalHostName()` for every single event if rDNS is enabled for an input. Java has no built-in cache for these requests and calls down to the operating system every single time. This introduces significant overhead and limits throughput, even with a caching resolver running on localhost.

In our tests we saw less than 1k events per second average throughput and a full processing buffer, while input and output buffers were empty and the CPU was idling. We investigated and found that all `processbufferprocessor` threads were blocked within `Inet6AddressImpl.getHostByAddr()` most of the time. Increasing pool size did not help much. After installing as local resolver with caching enabled, throughput was much better, but still clearly and severely bottle-necked by rDNS. Disabling rDNS more than doubled throughput.

This seems like an obvious and very rewarding opportunity to introduce an internal caching layer. There is usually a limited number of hosts logging to Graylog, so even a small cache with a short retention time would have an incredible hit-rate. I'm a little surprised that this was not done already.

Here is the stacktrace of the blocking calls:

at java.net.Inet6AddressImpl.getHostByAddr(java.base@11.0.12/Native Method)
at java.net.InetAddress$PlatformNameService.getHostByAddr(java.base@11.0.12/InetAddress.java:935)
at java.net.InetAddress.getHostFromNameService(java.base@11.0.12/InetAddress.java:659)
at java.net.InetAddress.getCanonicalHostName(java.base@11.0.12/InetAddress.java:632)
at org.graylog2.plugin.Tools.rdnsLookup(Tools.java:310)
at org.graylog2.inputs.codecs.SyslogCodec.parseHost(SyslogCodec.java:197)
at org.graylog2.inputs.codecs.SyslogCodec.parse(SyslogCodec.java:145)
at org.graylog2.inputs.codecs.SyslogCodec.decode(SyslogCodec.java:96)
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153)
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94)
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90)
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47)
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143)
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66)
at java.lang.Thread.run(java.base@11.0.12/Thread.java:829)

## Your Environment

* Graylog Version: 4.1.11
* Operating System: Debian 11

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.