elastic / elastic/docs-content

[Beats Inputs] Pipeline attribute in Beats inputs behavior and special handling

Open
#4,249 0 comments 0 reactions 0 assignees View on GitHub
Team:Ingest
Dominant language
No language data
Stars
47
Forks
261
Avg merge
3d 12h
Merged PRs (30d)
116

Description

Document the `pipeline` attribute in the inputs, such as:
- https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-log#_pipeline_16
- https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-filestream#_pipeline_9
- https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-udp#_pipeline_26
- TCP...
- All inputs exposing `pipeline`...

Will end up in a field called `@metadata.pipeline` when using Kafka & Logstash outputs.

In order to Elasticsearch to "execute" the pipeline, it will require special handling by the Logstash or anything consuming from Kafka to read `@metadata.pipeline` and pass it to the ES output.

E.g. for Logstash
```
if [@metadata][pipeline] {
elasticsearch {
hosts => ["http://localhost:9200"]
# cloud_id => "..."
data_stream => "true"
api_key => ""
data_stream => true
ssl_enabled => true
ssl_certificate_authorities => ""
pipeline => "%{[@metadata][pipeline]}" # <-------------------------------
}
} else {
elasticsearch {
hosts => ["http://localhost:9200"]
# cloud_id => "..."
data_stream => "true"
api_key => ""
data_stream => true
ssl_enabled => true
ssl_certificate_authorities => ""
}
}
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.