Graylog2 / Graylog2/graylog2-server

Stream search finds messages that doesn't belong to its index set

Open
#4,638 28 comments 0 reactions 1 assignee Claimed by @dennisoelkers View on GitHub
search to-verify
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior
Stream search should only find messages that corresponds to its index set

## Current Behavior
Stream search finds all messages matching its rules (even if they stored in different index sets, but it's true only for rotated indexes)

## Possible Solution
Remove stream tag from the message if that stream doesn't belong to its index set

## Steps to Reproduce
1. Create two streams with similar rule sets that belong to different index sets
2. Enable checkbox "Remove matches from 'All messages' stream" for the stream
3. Load messages that should go to both streams
4. Rotate their indexes
5. Search

## Context
We have two streams with similar rules: D2Requests and D2UserRequests (one of them is a subset of the other one), they belong to different index sets (first one -- default "graylog_", second -- custom with a longer lifetime "lt_graylog_").

![stream_d2r](https://user-images.githubusercontent.com/4670444/37024052-e4d17536-2138-11e8-9950-02fe838a2b50.PNG)
![stream_d2ur](https://user-images.githubusercontent.com/4670444/37024054-e4fac4fe-2138-11e8-835e-7edd52ba642f.PNG)

When you search in D2UserRequests stream there are duplicate messages but only for a rotated "default" indexes:

![stream_search](https://user-images.githubusercontent.com/4670444/37024100-0314e154-2139-11e8-8084-04692da64e08.PNG)

*Default index has been rotated at ~11:20.*

![stream_search_message1](https://user-images.githubusercontent.com/4670444/37024120-0d4d3a18-2139-11e8-9ca6-ba70d0a17b1a.PNG)

![stream_search_message2](https://user-images.githubusercontent.com/4670444/37024121-0d6d1c84-2139-11e8-8f18-518762a00bed.PNG)

*Same message, same stream -- different indexes.*

If you search "All messages" stream for messages that applied to D2UserRequests rules it finds nothing, so checkbox is working there.

## Your Environment
* Graylog Version: 2.4.3+2c41897
* Elasticsearch Version: 5.6.4
* MongoDB Version: 3.4.10
* Operating System: Linux 4.10.0-42-generic #46~16.04.1-Ubuntu SMP Mon Dec 4 15:57:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.