elastic / elastic/beats

Elasticsearch output: Add support for include_source_on_error query parameter

Open
#43,675 2 comments 0 reactions 0 assignees View on GitHub
Stalled Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

- Relates https://github.com/elastic/go-elasticsearch/issues/949
- Relates https://github.com/elastic/beats/pull/38767

With https://github.com/elastic/elasticsearch/pull/120725 Elasticsearch now supports a new query parameter include_source_on_error on bulk, created, index and update requests. When this option is set, ES will not include the document source in the error message in case of parsing errors.

In https://github.com/elastic/beats/pull/38767 we already directed the response content into a separate log file for inspection later, with nothing emitted directly in the Beat logs.

https://github.com/elastic/elasticsearch/pull/120725 would let us move away from the event log file and include an option in the config for whether response content should appear in the logs. This would eliminate the complexity of maintaining a separate event log but still run the risk that that Elastic Agent self-monitoring would ship sensitive or private event content to Fleet, so I'm not sure that it is an overall improvement since we already have the event log file.

Contributor guide

Open the contributing guide

Research direction

No file or test is named. Start by tracing the Elasticsearch output handling for bulk, create, index, and update requests, then review the linked Elasticsearch issue and Beats pull request for the expected parameter and logging behavior. Done means the requested support and its sensitive-content logging implications are settled and covered by appropriate tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, go
Domain
observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.