Elasticsearch output: Add support for include_source_on_error query parameter
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
- Relates https://github.com/elastic/go-elasticsearch/issues/949
- Relates https://github.com/elastic/beats/pull/38767
With https://github.com/elastic/elasticsearch/pull/120725 Elasticsearch now supports a new query parameter include_source_on_error on bulk, created, index and update requests. When this option is set, ES will not include the document source in the error message in case of parsing errors.
In https://github.com/elastic/beats/pull/38767 we already directed the response content into a separate log file for inspection later, with nothing emitted directly in the Beat logs.
https://github.com/elastic/elasticsearch/pull/120725 would let us move away from the event log file and include an option in the config for whether response content should appear in the logs. This would eliminate the complexity of maintaining a separate event log but still run the risk that that Elastic Agent self-monitoring would ship sensitive or private event content to Fleet, so I'm not sure that it is an overall improvement since we already have the event log file.
Contributor guide
Research direction
No file or test is named. Start by tracing the Elasticsearch output handling for bulk, create, index, and update requests, then review the linked Elasticsearch issue and Beats pull request for the expected parameter and logging behavior. Done means the requested support and its sensitive-content logging implications are settled and covered by appropriate tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, go
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100