Graylog2 / Graylog2/graylog2-server
Processing throughput drops drastically on non compilable grok patterns in rule
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## Expected Behavior
A rule with a grok pattern which does not exist can not be saved.
A grok pattern which is referenced in a rule should not be deletable
## Current Behavior
If a rule is created with an non existent grok pattern the rule can be saved.
If a message matches this rule a stacktrace gets written and the throughput drops drastically.
## Possible Solution
Rules which contain non existens GROK Pattern can not be saved.
GROK Pattern which are referenced in any rule can not be deleted.
## Steps to Reproduce (for bugs)
1.
Create following rule:
```
rule "non existend grok"
when
true
then
let result = grok(pattern: "%{NOTEXISTING:WORD} %{IPORHOST:client_ip}", value: to_string($message.message) ,only_named_captures: true);
set_fields(result);
end
```
2. Assign rule to a pipeline, connect pipeine to any stream which processes messages
3. Watch throughput drop drastically
4. Find following stacktrace over and over in your logfiles:
```
2023-05-30 10:42:04,378 ERROR: org.graylog2.grok.GrokPatternRegistry - Unable to load grok pattern %{NOTEXISTING:WORD} %{IPORHOST:client_ip} into cache
java.lang.IllegalArgumentException: No definition for key 'NOTEXISTING' found, aborting
at io.krakens.grok.api.GrokCompiler.compile(GrokCompiler.java:177) ~[graylog.jar:?]
at io.krakens.grok.api.GrokCompiler.compile(GrokCompiler.java:126) ~[graylog.jar:?]
at org.graylog2.grok.GrokPatternRegistry$GrokReloader.load(GrokPatternRegistry.java:132) ~[graylog.jar:?]
at org.graylog2.grok.GrokPatternRegistry$GrokReloader.load(GrokPatternRegistry.java:119) ~[graylog.jar:?]
at com.google.common.cache.CacheLoader$1.load(CacheLoader.java:193) ~[graylog.jar:?]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533) ~[graylog.jar:?]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282) ~[graylog.jar:?]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159) ~[graylog.jar:?]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049) ~[graylog.jar:?]
at com.google.common.cache.LocalCache.get(LocalCache.java:3966) ~[graylog.jar:?]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3989) ~[graylog.jar:?]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4950) ~[graylog.jar:?]
at org.graylog2.grok.GrokPatternRegistry.cachedGrokForPattern(GrokPatternRegistry.java:97) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.functions.strings.GrokMatch.evaluate(GrokMatch.java:63) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.functions.strings.GrokMatch.evaluate(GrokMatch.java:34) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.ast.expressions.FunctionExpression.evaluateUnsafe(FunctionExpression.java:63) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.ast.expressions.Expression.evaluate(Expression.java:36) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.ast.statements.VarAssignStatement.evaluate(VarAssignStatement.java:33) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.ast.statements.VarAssignStatement.evaluate(VarAssignStatement.java:22) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.evaluateStatement(PipelineInterpreter.java:395) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.executeRuleActions(PipelineInterpreter.java:380) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.evaluateStage(PipelineInterpreter.java:333) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.processForResolvedPipelines(PipelineInterpreter.java:277) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.process(PipelineInterpreter.java:149) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.process(PipelineInterpreter.java:105) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.handleMessage(ProcessBufferProcessor.java:158) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.dispatchMessage(ProcessBufferProcessor.java:128) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:98) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:49) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
```
## Context
This issues decreased our throughput drastically and lead to service outage
## Your Environment
* Graylog Version: 5.0.7
* Java Version: 17
* Opensearch Version: 2.7.0
* MongoDB Version: 5
* Operating System: RHEL
* Browser version: Firefox current ESR
Contributor guide
Assessment
This issue has not been assessed yet.