elastic / elastic/logstash

[FR/Bug] Improve logstash error messages when trying to index to a red index

Open
#18,884 0 comments 1 reaction 0 assignees View on GitHub
>bug enhancement status:needs-triage
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

At the moment, logstash logs a misleading error message when trying to write to a red index (Tested on `9.3`)

This is easily reproduced by creating an index with a routing requirement that cannot be satisfied :

```
PUT /red_index?timeout=5s
{
"settings": {
"index.routing.allocation.require._name": "non_existent_node"
}
}
```
Here's what logstash is logging :
```
{"level":"ERROR","loggerName":"logstash.outputs.elasticsearch","timeMillis":1774265617972,"thread":"[twitter.conf]>worker0","logEvent":{"message":"Attempted to send a bulk request but Elasticsearch appears to be unreachable or down","message":"Elasticsearch Unreachable: [[https://test-logger-d4f449.es.us-central1.gcp.cloud.es.io:443/_bulk?filter_path=errors,items.*.error,items.*.status][Manticore::SocketTimeout](https://test-logger-d4f449.es.us-central1.gcp.cloud.es.io/_bulk?filter_path=errors,items.*.error,items.*.status][Manticore::SocketTimeout)] Read timed out","exception":"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError","will_retry_in_seconds":2}}
{"level":"ERROR","loggerName":"logstash.outputs.elasticsearch","timeMillis":1774265620028,"thread":"[twitter.conf]>worker0","logEvent":{"message":"Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?)","message":"No Available connections","exception":"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError","will_retry_in_seconds":4}}
```

The bulk response returns a 200 but `errors` is true and `red_index` documents return a 503. For example :
```
{
"index": {
"_index": "red_index",
"_id": "A",
"status": 503,
"error": {
"type": "unavailable_shards_exception",
"reason": """[red_index][1] primary shard is not active Timeout: [5s], request: [BulkShardRequest [[red_index][1]] containing [index {[red_index][A], source[{ "title": "Doc A — should FAIL (red index)", "status": "fail" }]}]]"""
}
}
```

This looks like a regression / non propagation of the actual error message that should be available to Logstash and should be logged rather than logging a dead ES instance error message.

Contributor guide

Open the contributing guide

Research direction

Reproduce the problem with the shown red_index settings and inspect the logstash.outputs.elasticsearch handling of the /_bulk response. Use the returned errors and item status 503 as the reference, and consider the issue done when Logstash reports the unavailable_shards_exception details instead of an unreachable-Elasticsearch error.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.