elastic / elastic/docs-content
[Website]: Improve docs for using Logstash output with Fleet (_id situation)
- Dominant language
- No language data
- Stars
- 47
- Forks
- 261
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 116
Description
### Before you submit
- [x] This issue is about a documentation page, flow, or piece of content.
### Type of issue
Other
### What documentation page or section is affected
https://www.elastic.co/docs/reference/fleet/logstash-output
### What happened?
When using the Logstash output for Elastic Agent, in the output we recommend:
```
output {
if [@metadata][_id] {
elasticsearch {
...
document_id => "%{[@metadata][_id]}"
...
}
} else {
elasticsearch {
...
}
}
```
It is necessary because SOME but not all the integrations use `_id` to avoid duplication.
The drawback is this config duplicates the ES output and can lead to sub-optimal ingestion (due to bulk sizes might vary a lot in size).
A possible optimization for Logstash 8.12+ / or ES Output plugin 11.22.2 would be to leverage the undocumented `[@metadata][_ingest_document][id]`
The recommended config can become:
```
filter {
if [@metadata][_id] {
mutate { add_field => { "[@metadata][_ingest_document][id]" => "[@metadata][_id]" } }
}
}
output {
elasticsearch {
...
}
}
```
### Additional info
I would like @robbavey to review this.
The field `[@metadata][_ingest_document][id]` is not an actual public contract, but it might be useful to simplify the ES output.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.