Graylog2 / Graylog2/graylog2-server

Ingest old data into appropriate index file and/or discard

Open
#8,854 4 comments 0 reactions 0 assignees View on GitHub
feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior
When importing old log data (it might be from installing the sidecar on an existing server, or adding a new source of logs), old data should have the option of going into the approriately aged index file (or discarded if older than the oldest file) for a given index. Since data retention policy compliance is, as documented, down to the rotation policy of the index files, this would be required to ensure that data is deleted/archived as policy requires. I don't know whether it is even possible to do this or whether only the active index is writeable.

## Current Behavior
Currently all ingested data, regardless of age, goes into the active index file. If, say, you ingest old data from a server for log entries 5 months old, and you have a daily rotation strategy, retaining for 180 days (for a 6-month retention policy), those log entries won't be deleted until they are 11 months old, leading to a potential compliance breach if the log entries contain personal information.

## Possible Solution
It could be mitigated by client side filtering of what logs to send, so that only current logs are ingested and old logs ignored. But that might not always be desirable, or possible depending on the software sending the logs.
It may also be possible to use a pipeline to filter old data but again discarding the data may not be desirable (compliance may require that data is retained for the specified period).
An external tool could possible search for and move data between index files, or simply apply retention policies to delete data, but that would likely be slow and complicated as it would need to know how to identify the age of a record which might not have consistent fieldnames.

## Steps to Reproduce (for bugs)

## Context
I'm building a new graylog-based logging system, but want to identify and mitigate against all of the compliance issues before I start putting real data in and creating a risk of compliance breach,

## Your Environment

* Graylog Version: 3.3.5
* Java Version:
* Elasticsearch Version:
* MongoDB Version:
* Operating System:
* Browser version:

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.