elastic / elastic/beats

Automatically Preserve Ingest Pipeline Metadata from Elastic Agent

Open
#48,044 1 comment 0 reactions 0 assignees View on GitHub
needs_team
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 2h
Merged PRs (30d)
364

Description

## Summary

The Logstash Elasticsearch Output Plugin should automatically detect and apply the Elasticsearch Ingest Pipeline ID stored in the document's metadata (specifically [@metadata][pipeline]) when indexing documents.
This is would allow for seamless integration with Elastic Agent's custom pipeline configurations, as the Elastic Agent successfully attaches the target ingest pipeline ID to the document's metadata, typically as [@metadata][pipeline].

## The Issue

When the Elastic Agent sends data with a custom Ingest Pipeline ID (via the [Custom Logs Filestream Integration](https://www.elastic.co/docs/reference/integrations/filestream) in this case) to Logstash, the pipeline configuration is lost by default when indexing to Elasticsearch.

- Elastic Agent Behavior: The Agent attaches the intended pipeline ID to the document's metadata, typically as [@metadata][pipeline].
- Logstash Output Behavior: The current Logstash Elasticsearch Output Plugin does not implicitly check for and use this metadata field as the pipeline parameter in the API call.
- Workaround: Users need to manually add the following configuration to every relevant Logstash output block: `pipeline => "%{[@metadata][pipeline]}"`

## Impact
This results in configuration overhead, increases the risk of misconfiguration, and adds complexity, especially in environments routing multiple datastreams where some require this explicit setting and others do not.

## Request
Either Beats or Logstash could implement behavior that automatically detects and applies any custom ingest pipelines to the Elasticsearch output in Logstash.

## Benefits
This will reduce configuration overhead and allow the seamless integration of integrations and agents into the Logstash ecosystem, as well as reduce the potential for configuration errors.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.