Graylog2 / Graylog2/graylog2-server
Pipeline does not recognize streams are from different indices
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
I tried to create different views of the same log data by creating two streams assigned to two different index sets. When manipulating one of these streams using a processing pipeline, data in the other stream is being manipulated, too. The pipeline seems to ignore that it is connected to a single stream and processes all versions of a message in any available stream.
This bug report was written as advised over at:
https://community.graylog.org/t/anonymized-and-raw-views-of-same-logs-in-different-streams-possible/
## Expected Behavior
I expected the processing pipeline to only affect the stream it is connected to, especially given a stream in a seperate index set.
## Current Behavior
Instead of only manipulating the log data in the stream the pipeline is connected to, it affects all copies of the events in all index sets.
## Steps to Reproduce
1. create two new index sets
2. create two new streams, each attached to one of the new index sets
3. write identical stream rules for the new streams so that the data they contain is identical
4. create a new processing pipeline connected to only one of the new streams
5. in this processing pipeline, add a rule that causes a visible change to the log data
6. compare the data in the new streams; unexpectedly, the pipeline should affect both of them and the default index set
## Context
I am trying to create two views of the log data to set up a system that is GDPR compliant. The views of the logs meant to be used day by day should be anonymized while the raw data is available separately if tracking down an attacker or similar measures are neccessary.
## Your Environment
Message Processor Configuration:
1. Message Filter Chain (active)
2. Pipeline Processor (active)
3. AWS Instance Name Lookup (disabled)
4. GeoIP Resolver (active)
* Graylog Version: Graylog 2.4.6
* Elasticsearch Version: 5.6.10
* MongoDB Version: 3.2.11
* Operating System: Debian 4.9.110-3+deb9u2, 64bit
* Browser version: Firefox 60.1.0esr (32-Bit)
Contributor guide
Assessment
This issue has not been assessed yet.