Graylog2 / Graylog2/graylog2-server

Graylog drops messages nearly silently if elasticsearch is in indexing_pressure.memory.limit

Open
#13,997 3 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior

Dropped messages are visible in Indexing & Processing Failures or indexing gets retried
## Current Behavior

Messages get silently (?) dropped
## Possible Solution

## Steps to Reproduce (for bugs)

1. Reduce heap size in Elasticsearch
2. Give high pressure to elasticsearch for indexing
3. run in indexing_pressure.memory.limit
4. loose messages (?)
5. Observe "Bulk indexing finally successful (attempt \#x)." not getting log

## Logs
```
2022-11-18 09:29:10,594 ERROR: org.graylog2.indexer.messages.Messages - Couldn't bulk index 1000 messages.
java.util.concurrent.ExecutionException: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: ElasticsearchStatusException[Elasticsearch exception [type=es_rejected_execution_exception, reason=rejected execution of coordinating operation [co
ordinating_and_primary_bytes=853796361, replica_bytes=0, all_bytes=853796361, coordinating_operation_bytes=6710685, max_coordinating_and_primary_bytes=858993459]]];, errorDetails=[]}
at com.github.rholder.retry.Retryer$ExceptionAttempt.(Retryer.java:254) ~[graylog.jar:?]
at com.github.rholder.retry.Retryer.call(Retryer.java:163) ~[graylog.jar:?]
at org.graylog2.indexer.messages.Messages.runBulkRequest(Messages.java:225) [graylog.jar:?]
at org.graylog2.indexer.messages.Messages.bulkIndexRequests(Messages.java:148) [graylog.jar:?]
at org.graylog2.indexer.messages.Messages.bulkIndex(Messages.java:140) [graylog.jar:?]
at org.graylog2.indexer.messages.Messages.bulkIndex(Messages.java:120) [graylog.jar:?]
at org.graylog2.outputs.ElasticSearchOutput.writeMessageEntries(ElasticSearchOutput.java:103) [graylog.jar:?]
at org.graylog2.outputs.BlockingBatchedESOutput.flush(BlockingBatchedESOutput.java:117) [graylog.jar:?]
at org.graylog2.outputs.BlockingBatchedESOutput.writeMessageEntry(BlockingBatchedESOutput.java:98) [graylog.jar:?]
at org.graylog2.outputs.BlockingBatchedESOutput.write(BlockingBatchedESOutput.java:80) [graylog.jar:?]
at org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:193) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.graylog2.indexer.ElasticsearchException: ElasticsearchException[An error occurred: ]; nested: ElasticsearchStatusException[Elasticsearch exception [type=es_rejected_execution_exception, reason=rejected execution of coordinating operation [coordinating_and_p
rimary_bytes=853796361, replica_bytes=0, all_bytes=853796361, coordinating_operation_bytes=6710685, max_coordinating_and_primary_bytes=858993459]]];
at org.graylog.storage.elasticsearch7.MessagesAdapterES7.runBulkRequest(MessagesAdapterES7.java:181) ~[?:?]
at org.graylog.storage.elasticsearch7.MessagesAdapterES7.bulkIndexChunked(MessagesAdapterES7.java:126) ~[?:?]
at org.graylog2.indexer.messages.ChunkedBulkIndexer.index(ChunkedBulkIndexer.java:43) ~[graylog.jar:?]
at org.graylog.storage.elasticsearch7.MessagesAdapterES7.bulkIndex(MessagesAdapterES7.java:108) ~[?:?]
at org.graylog2.indexer.messages.Messages.lambda$runBulkRequest$6(Messages.java:225) ~[graylog.jar:?]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[graylog.jar:?]
at com.github.rholder.retry.Retryer.call(Retryer.java:160) ~[graylog.jar:?]
... 15 more
Caused by: org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: An error occurred:
at org.graylog.storage.elasticsearch7.ElasticsearchClient.exceptionFrom(ElasticsearchClient.java:151) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:111) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:104) ~[?:?]
at org.graylog.storage.elasticsearch7.MessagesAdapterES7.runBulkRequest(MessagesAdapterES7.java:174) ~[?:?]
at org.graylog.storage.elasticsearch7.MessagesAdapterES7.bulkIndexChunked(MessagesAdapterES7.java:126) ~[?:?]
at org.graylog2.indexer.messages.ChunkedBulkIndexer.index(ChunkedBulkIndexer.java:43) ~[graylog.jar:?]
at org.graylog.storage.elasticsearch7.MessagesAdapterES7.bulkIndex(MessagesAdapterES7.java:108) ~[?:?]
at org.graylog2.indexer.messages.Messages.lambda$runBulkRequest$6(Messages.java:225) ~[graylog.jar:?]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[graylog.jar:?]
at com.github.rholder.retry.Retryer.call(Retryer.java:160) ~[graylog.jar:?]
... 15 more
Caused by: org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=es_rejected_execution_exception, reason=rejected execution of coordinating operation [coordinating_and_primary_bytes=853796361, replica_bytes=0, all_bytes=853796361, coordinating_operation_bytes=6710685, max_coordinating_and_primary_bytes=858993459]]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:533) ~[?:?]
at org.graylog.storage.elasticsearch7.MessagesAdapterES7.lambda$runBulkRequest$2(MessagesAdapterES7.java:174) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:109) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:104) ~[?:?]
at org.graylog.storage.elasticsearch7.MessagesAdapterES7.runBulkRequest(MessagesAdapterES7.java:174) ~[?:?]
at org.graylog.storage.elasticsearch7.MessagesAdapterES7.bulkIndexChunked(MessagesAdapterES7.java:126) ~[?:?]
at org.graylog2.indexer.messages.ChunkedBulkIndexer.index(ChunkedBulkIndexer.java:43) ~[graylog.jar:?]
at org.graylog.storage.elasticsearch7.MessagesAdapterES7.bulkIndex(MessagesAdapterES7.java:108) ~[?:?]
at org.graylog2.indexer.messages.Messages.lambda$runBulkRequest$6(Messages.java:225) ~[graylog.jar:?]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[graylog.jar:?]
at com.github.rholder.retry.Retryer.call(Retryer.java:160) ~[graylog.jar:?]
... 15 more
Suppressed: org.graylog.shaded.elasticsearch7.org.elasticsearch.client.ResponseException: method [POST], host [http://fancyhost:9200], URI [/_bulk?timeout=1m], status line [HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"es_rejected_execution_exception","reason":"rejected execution of coordinating operation [coordinating_and_primary_bytes=853796361, replica_bytes=0, all_bytes=853796361, coordinating_operation_bytes=6710685, max_coordinating_and_primary_bytes=858993459]"}],"type":"es_rejected_execution_exception","reason":"rejected execution of coordinating operation [coordinating_and_primary_bytes=853796361, replica_bytes=0, all_bytes=853796361, coordinating_operation_bytes=6710685, max_coordinating_and_primary_bytes=858993459]"},"status":429}
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:272) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:533) ~[?:?]
at org.graylog.storage.elasticsearch7.MessagesAdapterES7.lambda$runBulkRequest$2(MessagesAdapterES7.java:174) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:109) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:104) ~[?:?]
at org.graylog.storage.elasticsearch7.MessagesAdapterES7.runBulkRequest(MessagesAdapterES7.java:174) ~[?:?]
at org.graylog.storage.elasticsearch7.MessagesAdapterES7.bulkIndexChunked(MessagesAdapterES7.java:126) ~[?:?]
at org.graylog2.indexer.messages.ChunkedBulkIndexer.index(ChunkedBulkIndexer.java:43) ~[graylog.jar:?]
at org.graylog.storage.elasticsearch7.MessagesAdapterES7.bulkIndex(MessagesAdapterES7.java:108) ~[?:?]
at org.graylog2.indexer.messages.Messages.lambda$runBulkRequest$6(Messages.java:225) ~[graylog.jar:?]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[graylog.jar:?]
at com.github.rholder.retry.Retryer.call(Retryer.java:160) ~[graylog.jar:?]

```

## Context

## Your Environment

* Graylog Version: 4.3.9
* Java Version: 11
* Elasticsearch Version: 7.10.2
* MongoDB Version: 4.4
* Operating System: RHEL
* Browser version: %

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.