Graylog2 / Graylog2/graylog2-server

Can't update pipeline rules in content pack

Open
#7,745 4 comments 1 reaction 0 assignees View on GitHub
content-packs feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

excerpted from [forum post](https://community.graylog.org/t/cant-update-a-content-pack/14573)

I created a content pack to distribute pipeline rules to the various clusters I have. I updated some of the rules, so I figured I could just update the revision number, upload that everywhere, and be all set.

I was able to upload the new content pack, but when I select “Install” from the Actions list on the new revision I get a 500 error. In the server logs I find the following error:
`Caused by: org.graylog2.contentpacks.exceptions.DivergingEntityConfigurationException: Different pipeline rule sources for pipeline rule with name "upload"`
## Expected Behavior
I expected that making a new revision to my content pack would allow me to update the pipeline rules in place. I assumed it would just overwrite the existing rules since they have the same names.

## Current Behavior
When installing a content pack that contains a pipeline rule with the same name as an existing rule, there is a check to see if they contain the same "source." If they do not, an error is thrown.

```
2020-03-20T22:12:18.724Z DEBUG [PipelineRuleFacade] Expected source for rule "upload":
rule "upload"
when
has_field("source") AND contains(to_string($message.source), "pod") AND
((has_field("file") AND contains(to_string($message.file), "upload")) OR
(has_field("filebeat_source") AND contains(to_string($message.filebeat_source), "upload")))
then
let result = grok("%{GROK_PATTERN}", to_string($message.message));
set_fields(result);
end

Actual source:
rule "upload"
when
has_field("source") AND contains(to_string($message.source), "pod") AND
has_field("file") AND contains(to_string($message.file), "upload")
then
let result = grok("%{GROK_PATTERN}", to_string($message.message));
set_fields(result);
end
2020-03-20T22:12:18.724Z ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
org.graylog2.contentpacks.exceptions.ContentPackException: Failed to install content pack
at org.graylog2.contentpacks.ContentPackService.installContentPack(ContentPackService.java:158) ~[graylog.jar:?]
at org.graylog2.contentpacks.ContentPackService.installContentPack(ContentPackService.java:99) ~[graylog.jar:?]
at org.graylog2.rest.resources.system.contentpacks.ContentPackResource.installContentPack(ContentPackResource.java:294) ~[graylog.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_152]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_152]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_152]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_152]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
Caused by: org.graylog2.contentpacks.exceptions.DivergingEntityConfigurationException: Different pipeline rule sources for pipeline rule with name "upload"
at org.graylog2.contentpacks.facades.PipelineRuleFacade.compareRuleSources(PipelineRuleFacade.java:151) ~[graylog.jar:?]
at org.graylog2.contentpacks.facades.PipelineRuleFacade.findExisting(PipelineRuleFacade.java:140) ~[graylog.jar:?]
at org.graylog2.contentpacks.facades.PipelineRuleFacade.findExisting(PipelineRuleFacade.java:126) ~[graylog.jar:?]
at org.graylog2.contentpacks.ContentPackService.installContentPack(ContentPackService.java:131) ~[graylog.jar:?]
... 29 more
```

## Possible Solution
Provide a "Yes, I know what I'm doing" checkbox to allow overwrites of existing pipeline rules.
I noticed that the Grok Pattern and Lookup Table facades also contain similar comparison code, so I would assume such an option would be useful with those as well.

## Steps to Reproduce (for bugs)

1. Create a content pack with a pipeline rule.
2. Install content pack on a Graylog node.
3. Change the pipeline rule's source code, but not its name.
4. Increase the revision number of the content pack.
5. Try to install the new version of the content pack on the same Graylog node.

## Context
I'm trying to use content packs to distribute pipeline rules to multiple clusters. My workaround is to manually update each pipeline rule in each cluster, but that does not scale well and is time consuming.

## Your Environment

* Graylog Version: Graylog 3.1.4+1149fe1
* Elasticsearch Version: ES: 6.8.3
* MongoDB Version: 4.0.16
* Operating System: macOS Mojave: 10.14.6
* Browser version: Google Chrome: 78.0.3904.70

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.