Graylog2 / Graylog2/graylog2-server

When rotating main index, rolling-file logger can eat all ressources until java out of memory and graylog is down

Open
#12,075 1 comment 0 reactions 0 assignees View on GitHub
bug to-verify
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior
logging to the file-rolling appender server.log file should not impair graylog perfomances.

## Current Behavior

A some point main index is rotated. In case of a lot of indexing errors (we have arround 40k indexing errors per day) it seems all past errors are thrown again (reindexing ?) (we got more than 2M in our case ) then the rolling file log4j2 appender litteraly got a DOS and consumed all java resources until Graylog died because out of memory. The UI cannot be reached.

The issue started immediately after :
```
2022-02-01T09:46:18.903+01:00 INFO [AbstractRotationStrategy] Deflector index (index set ) should be rotated, Pointing deflector to new index now!
2022-02-01T09:46:18.906+01:00 INFO [MongoIndexSet] Cycling from to .
2022-02-01T09:46:18.908+01:00 INFO [MongoIndexSet] Creating target index .
2022-02-01T09:46:19.152+01:00 INFO [MongoIndexSet] Waiting for allocation of index .
2022-02-01T09:46:19.158+01:00 INFO [MongoIndexSet] Index has been successfully allocated.
2022-02-01T09:46:19.160+01:00 INFO [MongoIndexSet] Pointing index alias to new index .
2022-02-01T09:46:19.197+01:00 INFO [SystemJobManager] Submitted SystemJob <6cb827c0-833b-11ec-ba59-02384db95920> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
2022-02-01T09:46:19.197+01:00 INFO [MongoIndexSet] Successfully pointed index alias to index .

```

The logs are then full of
```
"ERROR [MessagesAdapterES7] Failed to index "
```
and the detailed messages:

```
[0]: index [graylog_4], type [_doc], id [71059070-835d-11ec-9adc-02384db95920], message [ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse field [node_module] of type [date] in document with id '71059070-835d-11ec-9adc-02384db95920'. Preview of field's value: '[318]']]; nested: ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=failed to parse date field [[318]] with format [strict_date_optional_time||epoch_millis]]]; nested: ElasticsearchException[Elasticsearch exception [type=date_time_parse_exception, reason=Failed to parse with all enclosed parsers]];]
```
and we see :
```
INFO [Buffers] Waiting for buffers to drain. (0i/0p/24996o)
```

until we ultimately crash with :

```
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
```

## Possible Solution
we tried to allocate more memory to the java heap (1GB-> 4GB) for graylog with no success.

we tried to filter out the messages with filters in log4j2.xml but none worked :

```


```

the only way to get graylog running again was to disable completely the appender :
```

```

The log file appender should maybe avoid logging the ElasticsearchException by default.
But ultimately the log appender should not be able to impair graylog.

## Steps to Reproduce (for bugs)
Index a lot of messages with a lot of indexing error.
Rotate main index.

## Your Environment

* Graylog Version: 4.2.5
* Java Version: openjdk-17
* Elasticsearch Version: 7.10.2
* MongoDB Version: 4.0.28
* Operating System: ubuntu 20.04

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.