Calls to mutate make start up very slow
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 63
Description
- Version: LogStash v5.4.1
- Operating System: Ubuntu 16.04 inside a Docker 17.03-ce container on NixOS 17.03
- Config File (if you have sensitive info, please remove it): See below
- Steps to Reproduce: Start LogStash
This is likely related to #6460 and/or #6024
Note that I am providing LogStash with this `jvm.options` file:
```
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-Dcompile.invokedynamic=true
-Xms2g
-Xmx5g
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-XX:+DisableExplicitGC
```
We have a filter configuration similar to this (which is inefficient but bear with me):
```
filter {
if "arobat.exe" == [process_name] and "Adobe Systems" in [binary][digsig_publisher] { mutate { add_field => { "whitelisted" => true } } }
if "autonomyirm.exe" == [process_name] and "Hewlett-Packard" in [binary][company_name] { mutate { add_field => { "whitelisted" => true } } }
if "ccsvchst.exe" == [process_name] and "Symantec" in [binary][company] { mutate { add_field => { "whitelisted" => true } } }
if "ccsvchst.exe" == [process_name] and "symantec" in [process][cmdline] and "symantec" in [binary][digsig_publisher] { mutate { add_field => { "whitelisted" => true } } }
if "arobat.exe" == [process_name] and "Adobe Systems" in [binary][digsig_publisher] { mutate { add_field => { "whitelisted" => true } } }
if "autonomyirm.exe" == [process_name] and "Hewlett-Packard" in [binary][company_name] { mutate { add_field => { "whitelisted" => true } } }
... # Imagine 1200 of these, all similar in structure
}
```
Startup up LogStash with this configuration takes roughly 10 minutes.
I ran a few things against the Docker container, it seems the JVM is GC'ing like crazy, lots and lots of little objects are created and garbage collected.
A single CPU core constantly sits at 100% (presumably because of the GC activity) during the 10-minute startup.
Changing the configuration to a giant disjunction like so (logically identical if we ignore short-circuiting at runtime):
```
filter {
if (
("arobat.exe" == [process_name] and "Adobe Systems" in [binary][digsig_publisher]) or
("autonomyirm.exe" == [process_name] and "Hewlett-Packard" in [binary][company_name]) or
("ccsvchst.exe" == [process_name] and "Symantec" in [binary][company]) or
... # Image 1200 of these
) {
mutate {
add_field => { "whitelisted" => "true" }
}
}
}
```
Now startup takes less than 30 seconds.
Based on these observations it seems that calls to `mutate` (and possibly plugins in general?) really slows down LogStash startup.
Given that all our `mutate` steps are identical, some sort of optimization could have happened here. Or at the very least, start up should not be taking 10 minutes.
Contributor guide
Assessment
This issue has not been assessed yet.