Elasticsearch sink: log the original event content on non-retryable/dropped failures (like Logstash's `:action=>` field)

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

When the Elasticsearch sink drops an event due to a non-retryable error (e.g. document_parsing_exception, mapper_parsing_exception), there is currently no way to recover which original event was dropped, or its content, from Vector's own logs/telemetry.

Logstash's elasticsearch output plugin logs the full original document alongside the ES error in this situation, e.g.:

[WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["create", {:_id=>nil, :_index=>"...", :routing=>nil}, {"kubernetes.pod.name"=>"...", "message"=>"...", ...full original document...}], :response=>{"create"=>{"_index"=>"...", "status"=>400, "error"=>{"type"=>"document_parsing_exception", "reason"=>"..."}}}}

This lets an operator (or the team that owns the log source) directly identify which specific log line was dropped and why, by searching their own upstream logs/application output for the dropped content.

Vector's elasticsearch sink already holds the equivalent data in memory - ElasticsearchRequest.original_events: Vec<ProcessedEvent> (src/sinks/elasticsearch/service.rs) is populated specifically to support reconstructing partial-retry requests (src/sinks/elasticsearch/retry.rs) - but it is never passed to any tracing log call, at any verbosity level (checked debug!/trace!/error!/warn!/info! across the whole sink and the shared retry module). So there is currently no way to get this information out of Vector at all, regardless of log level/--internal-log-rate-limit/-vvv.

Without this, diagnosing "why did my application's log line never make it to Elasticsearch" requires guessing based on the (much narrower) per-field error text in reason (e.g. "Preview of field's value" only shows the one field that broke parsing, not the rest of the document), which usually isn't enough to positively identify the dropped event.

Attempted Solutions

• Raising --internal-log-rate-limit - doesn't help, this data isn't logged at any rate, not just rate-limited.
• Parsing the ES bulk response body embedded in the sink's "Response contained errors." log line - gives the target _index/_id/error.type/error.reason, which is enough to attribute failures to an owning component/team, but still doesn't recover the original event content.

Proposal

When an Elasticsearch sink request is finally dropped (non-retryable error, or retries exhausted), log the original event(s) for at least the specific items that failed - similar in spirit to Logstash's :action=> field. original_events is already available at the point where the failure is determined; this would mean adding a log call (likely gated behind a lower log level such as debug or trace, given the potential size/frequency, and/or behind an opt-in config flag given this could contain sensitive data) that includes the dropped event's fields alongside the existing reason.

This doesn't need to be on by default or unconditional - even a debug-level or explicitly opt-in log line would be a major improvement over the current "no path to this data at all."

References

No response

Version

0.51.1

Start in src/sinks/elasticsearch/service.rs by tracing ElasticsearchRequest.original_events into the failure path, then read src/sinks/elasticsearch/retry.rs to understand partial-retry reconstruction. Done means non-retryable or exhausted failures can expose the relevant original event content through an appropriately gated log path, with sensitive-data and volume concerns addressed.