fluent / fluent/fluentd

Add error messages to events getting routed to the @ERROR label

Open
#3,823 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

### Is your feature request related to a problem? Please describe.

Hi,

we use Fluentd for receiving various log messages in syslog or json format from our users and pushing them to OpenSearch indices. This works nicely and is pretty reliable. If for some reason OpenSearch rejects a record (e.g., due to a parsing exception), we see a detailed error message in the fluentd logs giving additional information on the rejection. This information is quite helpful for resolving the problem.

In addition to the error message in the fluentd logs an error event gets emitted to the `@ERROR` label for further processing (related [docs](https://docs.fluentd.org/configuration/config-file#error-label)). Our idea is to take the error message from the emitted error event and forward it to a dedicated index in OpenSearch. This way we could make rejection errors visible to our users without forwarding all Fluentd logs to the OpenSearch cluster.

Unfortunately, the emitted error events do not include the error message and/or stacktrace. In this [code section](https://github.com/fluent/fluentd/blob/c6b2fe2e2de91a99385ceee62a247f3aa5262bc9/lib/fluent/root_agent.rb#L348-L349) the `error_info` object gets printed while the tag, timestamp and record get emitted to the `@ERROR` label. In result it is not possible to forward the error message/stacktrace as part of the error event to OpenSearch or any other output.

### Describe the solution you'd like

I can think of following solution:
The `error_info` object could be added to a dedicated field in the record being part of the emitted error event. The field could be optional and configurable to avoid collisions with existing fields in the record. This way it would be possible to extract and route the error message/stacktrace and maybe additional information from the record to the output.

### Describe alternatives you've considered

Technically, it is possible to forward the fluentd logs to our OpenSearch instances. This would include the error message we would like to see. However, this approach requires additional filtering and we would loose the option to forward the rejected record side-by-side with the related error message which can be quite helpful when debugging errors.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.