Graylog2 / Graylog2/graylog2-server

Ability to send system logs into ES index

Open
#5,925 2 comments 0 reactions 0 assignees View on GitHub
feature infrastructure triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

I've started to notice signal-to-noise issues with the system overview page's centralized node logs. Every input change or general graylog cluster change results in at least 6 pages of repeating logs as each node does the same action. This makes paging through looking for actual information irksome.

## Expected Behavior
There should be an option to tee off the graylog system logs to the backend ES in a dedicated index set.

## Current Behavior
Graylog system logs are stored in mongodb.
35 graylog nodes * 3 messages per input change, deleting for example: STOPPING / STOPPED / TERMINATED = 105 log events when I delete an input, resulting in pages and pages of that event repeating.
![image](https://user-images.githubusercontent.com/568957/57089285-be649580-6cc9-11e9-885a-8d87bb9094cd.png)

## Possible Solution
I understand it could be problematic to ONLY store system logs into ES in the event the backend ES cluster is having problems. Potentially use mongodb as primary and also index into a dedicated index for graylog_system, allowing you to use the graylog ecosystem to explore its own logs.

Another option would be to summarize identical events across the cluster as something like:
> 2019-05-02T10:31:46-05:00 [35 nodes] Input [Raw/Plaintext TCP/5ccb0d543b3e1e7e1bbeb274] is now STARTING

## Steps to Reproduce (for bugs)
Make a lot of graylog nodes. Do stuff. Look at the system overview page.

## Context
I routinely check for things like indexes rollover rate and the easiest location to eyeball that is in the system overview page. It might be nice to also be able to use all the graylog feature set to review its own logs. Long term storage of graylog status logs also might not be best done in the configuration mongodb.

* Graylog Version: 3.0.1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.