Graylog2 / Graylog2/graylog2-server

Pipeline does not recognize streams are from different indices

Open
#5,016 3 comments 0 reactions 0 assignees View on GitHub
bug processing triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

I tried to create different views of the same log data by creating two streams assigned to two different index sets. When manipulating one of these streams using a processing pipeline, data in the other stream is being manipulated, too. The pipeline seems to ignore that it is connected to a single stream and processes all versions of a message in any available stream.
This bug report was written as advised over at:
https://community.graylog.org/t/anonymized-and-raw-views-of-same-logs-in-different-streams-possible/

## Expected Behavior

I expected the processing pipeline to only affect the stream it is connected to, especially given a stream in a seperate index set.

## Current Behavior

Instead of only manipulating the log data in the stream the pipeline is connected to, it affects all copies of the events in all index sets.

## Steps to Reproduce

1. create two new index sets
2. create two new streams, each attached to one of the new index sets
3. write identical stream rules for the new streams so that the data they contain is identical
4. create a new processing pipeline connected to only one of the new streams
5. in this processing pipeline, add a rule that causes a visible change to the log data
6. compare the data in the new streams; unexpectedly, the pipeline should affect both of them and the default index set

## Context

I am trying to create two views of the log data to set up a system that is GDPR compliant. The views of the logs meant to be used day by day should be anonymized while the raw data is available separately if tracking down an attacker or similar measures are neccessary.

## Your Environment

Message Processor Configuration:
1. Message Filter Chain (active)
2. Pipeline Processor (active)
3. AWS Instance Name Lookup (disabled)
4. GeoIP Resolver (active)

* Graylog Version: Graylog 2.4.6
* Elasticsearch Version: 5.6.10
* MongoDB Version: 3.2.11
* Operating System: Debian 4.9.110-3+deb9u2, 64bit
* Browser version: Firefox 60.1.0esr (32-Bit)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.