Automatically Preserve Ingest Pipeline Metadata from Elastic Agent
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 364
Description
## Summary
The Logstash Elasticsearch Output Plugin should automatically detect and apply the Elasticsearch Ingest Pipeline ID stored in the document's metadata (specifically [@metadata][pipeline]) when indexing documents.
This is would allow for seamless integration with Elastic Agent's custom pipeline configurations, as the Elastic Agent successfully attaches the target ingest pipeline ID to the document's metadata, typically as [@metadata][pipeline].
## The Issue
When the Elastic Agent sends data with a custom Ingest Pipeline ID (via the [Custom Logs Filestream Integration](https://www.elastic.co/docs/reference/integrations/filestream) in this case) to Logstash, the pipeline configuration is lost by default when indexing to Elasticsearch.
- Elastic Agent Behavior: The Agent attaches the intended pipeline ID to the document's metadata, typically as [@metadata][pipeline].
- Logstash Output Behavior: The current Logstash Elasticsearch Output Plugin does not implicitly check for and use this metadata field as the pipeline parameter in the API call.
- Workaround: Users need to manually add the following configuration to every relevant Logstash output block: `pipeline => "%{[@metadata][pipeline]}"`
## Impact
This results in configuration overhead, increases the risk of misconfiguration, and adds complexity, especially in environments routing multiple datastreams where some require this explicit setting and others do not.
## Request
Either Beats or Logstash could implement behavior that automatically detects and applies any custom ingest pipelines to the Elasticsearch output in Logstash.
## Benefits
This will reduce configuration overhead and allow the seamless integration of integrations and agents into the Logstash ecosystem, as well as reduce the potential for configuration errors.
Contributor guide
Assessment
This issue has not been assessed yet.