elastic / elastic/docs-content

[Website]: Improve docs for using Logstash output with Fleet (_id situation)

Open
#7,854 1 comment 1 reaction 1 assignee Claimed by @vishaangelova View on GitHub
source:web Team:SKI
Dominant language
No language data
Stars
47
Forks
261
Avg merge
3d 12h
Merged PRs (30d)
116

Description

### Before you submit

- [x] This issue is about a documentation page, flow, or piece of content.

### Type of issue

Other

### What documentation page or section is affected

https://www.elastic.co/docs/reference/fleet/logstash-output

### What happened?

When using the Logstash output for Elastic Agent, in the output we recommend:
```
output {

if [@metadata][_id] {
elasticsearch {
...
document_id => "%{[@metadata][_id]}"
...
}
} else {
elasticsearch {
...
}
}
```

It is necessary because SOME but not all the integrations use `_id` to avoid duplication.

The drawback is this config duplicates the ES output and can lead to sub-optimal ingestion (due to bulk sizes might vary a lot in size).

A possible optimization for Logstash 8.12+ / or ES Output plugin 11.22.2 would be to leverage the undocumented `[@metadata][_ingest_document][id]`

The recommended config can become:
```
filter {
if [@metadata][_id] {
mutate { add_field => { "[@metadata][_ingest_document][id]" => "[@metadata][_id]" } }
}
}
output {
elasticsearch {
...
}
}
```

### Additional info

I would like @robbavey to review this.

The field `[@metadata][_ingest_document][id]` is not an actual public contract, but it might be useful to simplify the ES output.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.