Graylog2 / Graylog2/graylog2-server
Event Notification Backlog Issue
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
Enabling and setting a Message Backlog > 1 in the Notification settings appears to have no effect on the resulting e-mail notification. This occurs even when creating a new Event Definition and Notification in Graylog v3.3.9. No error messages are thrown in the ElasticSearch or Graylog application logs.
## Expected Behavior
A number of messages returned by the event filter, up to a count of $message_backlog, should be included in the e-mail notification.
## Current Behavior
A single event matched by the filter is included in the e-mail notification.
## Possible Solution
## Steps to Reproduce (for bugs)
1. Create a new event definition which could have multiple matches in a given time period. Ensure that in the "Notifications" section, Message Backlog is enabled and set to a value greater than 1 (we used 50).
2. Create an e-mail notification and associate it with this event. The default Body Template as follows was used:
```
--- [Event Definition] ---------------------------
Title: ${event_definition_title}
Description: ${event_definition_description}
Type: ${event_definition_type}
--- [Event] --------------------------------------
Timestamp: ${event.timestamp}
Message: ${event.message}
Source: ${event.source}
Key: ${event.key}
Priority: ${event.priority}
Alert: ${event.alert}
Timestamp Processing: ${event.timestamp}
Timerange Start: ${event.timerange_start}
Timerange End: ${event.timerange_end}
Fields:
${foreach event.fields field} ${field.key}: ${field.value}
${end}
${if backlog}
--- [Backlog] ------------------------------------
Last messages accounting for this alert:
${foreach backlog message}
${message}
${end}
${end}
```
3. Wait for Event to trigger an e-mail notification.
4. A single backlog message is included in the e-mail:
```
--- [Event Definition] ---------------------------
Title: Testing
Description:
Type: aggregation-v1
--- [Event] --------------------------------------
Timestamp: 2020-12-01T17:32:44.000Z
Message: Testing
Source: graylog-prod
Key:
Priority: 2
Alert: true
Timestamp Processing: 2020-12-01T17:32:44.000Z
Timerange Start:
Timerange End:
Fields:
--- [Backlog] ------------------------------------
Last messages accounting for this alert:
{index=rsyslog_97, message=, timestamp=2020-12-01T17:32:44.000Z, source=>, stream_ids=[], fields={gl2_accounted_message_size=374, level=5, gl2_remote_ip=, gl2_remote_port=, gl2_message_id=, gl2_source_node=, gl2_source_input=, facility=local0}, id=}
```
## Context
This issue makes it a more difficult to properly respond to alerts. We either need to accept a flood of e-mail alerts each time an event is matched (I.E. a single e-mail per event), or manually search to see if there are other events when a notification fires. The ability to see that there are multiple matched alerts in a single event notification would provide additional welcome context as to potential severity or impact of an Event that a single match might not.
## Your Environment
* Graylog Version: 3.3.9
* Java Version: 1.7.0
* Elasticsearch Version:
* MongoDB Version: 4.0.21
* Operating System: Amazon Linux
* Browser version: Chrome 87.0.4280.67
Thanks & happy to provide any additional information/perform any troubleshooting needed!
Contributor guide
Assessment
This issue has not been assessed yet.