elastic / elastic/logstash

Improved error logging - add more details including index/data stream

Open
#18,465 1 comment 1 reaction 0 assignees View on GitHub
enhancement status:needs-triage
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

I have a Logstash pipeline that gets logs from Elasticsearch data streams and uploads them to AWS S3 bucket.
Recently, I've added one more data stream name to the `index` field in the `input` section, and started getting errors that the user does not have enough permissions. But the problem is that the user has enough permissions because when I run the pipeline separately for new and old data streams, everything works well.

Original log message
```
[2025-12-04T15:10:07,921][ERROR][logstash.inputs.elasticsearch.searchafter][my-logs][10655218...]
Tried search_after paginated search unsuccessfully
{:message=>"[403] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/read/search] is unauthorized for user [my-logstash-user] with effective roles [eck_logstash_user_role,monitoring,writer], this action is granted by the index privileges [read,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/read/search] is unauthorized for user [my-logstash-user] with effective roles [eck_logstash_user_role,monitoring,writer], this action is granted by the index privileges [read,all]\"},\"status\":403}", :cause=>nil}
```

Message from above, but formatted:
```json
{
"error": {
"root_cause": [
{
"type": "security_exception",
"reason": "action [indices:data/read/search] is unauthorized for user [my-logstash-user] with effective roles [eck_logstash_user_role,monitoring,writer], this action is granted by the index privileges [read,all]"
}
],
"type": "security_exception",
"reason": "action [indices:data/read/search] is unauthorized for user [my-logstash-user] with effective roles [eck_logstash_user_role,monitoring,writer], this action is granted by the index privileges [read,all]"
},
"status": 403
}
```

Logstash pipeline config:
```ruby
input {
elasticsearch {
hosts => ["${ECK_ES_HOSTS}"]
user => "${ECK_ES_USER}"
password => "${ECK_ES_PASSWORD}"
index => "svc1*,svc2*,svc3*,svc4*"
query => '{ "query": { "bool": { "filter": [{ "range": { "@timestamp": { "gte": "now-1d/d", "lte": "now-1d/d" }}}, { "match_phrase": { "tags": "my" } }] }}, "sort": [{ "@timestamp": { "order": "asc" }}] }'
size => 5000
schedule => "0 1 * * *"
}
}

filter {
mutate {
remove_field => ["kubernetes"]
}
}

output {
s3 {
id => "my-logs"
bucket => "my-logs"
region => "eu-central-1"
prefix => "raw/app=%{[app]}/year=%{+YYYY}/month=%{+MM}/day=%{+dd}"
encoding => "gzip"
canned_acl => "bucket-owner-full-control"
rotation_strategy => "size_and_time"
size_file => 268435456 # 256MB in bytes
time_file => 15
codec => "json_lines"
temporary_directory => "${HOME}/data/my-logs"
}
}
```
When I run this pipeline with `index => "svc1*,svc2*,svc3*"` and `index => "svc4*"`, it works well, but when I specify all these data streams together, it fails with the error.

Please add more details to the error message to solve the issue I'm facing with.

Similar request but for Kibana - https://github.com/elastic/kibana/issues/126255.

Contributor guide

Open the contributing guide

Research direction

Start at the `logstash.inputs.elasticsearch.searchafter` error path shown in the log and trace how the failed search request is reported when multiple index patterns are used. The work is done when the error identifies the relevant index or data stream and the existing Elasticsearch input tests cover the added details; no test file is named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, elasticsearch
Domain
data-engineering, observability-sre
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.