fluent / fluent/fluentd

Add support for metadata in Fluentd event structure

Open
#4,141 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

### Is your feature request related to a problem? Please describe.

Currently, Fluentd handles the following events structure:

```
[TAG, TIMESTAMP, {RECORD}]
```

We wanted to change to handle metadata in Fluentd event structure:

```
[TAG, [TIMESTAMP, {METADATA}], {RECORD}]
```

Note that, in the above structure, timestamp is already one of the metadata structure and other metadata should be handled in msgpack map that is represented Hash in Ruby.

### Describe the solution you'd like

Handle metadata requirements in Fluentd.

### Describe alternatives you've considered

The minimum solution is just handled as normal Forwarded events with metadata from fluent-bit:

From fluent-bit, events metadata should be sent as:

```
[TAG, TIMESTAMP, {RECORD}, {metadata key/val in OPTION}]
```

And metadata part should be stored in a special key. However, supporting metadata in the second part of the event should be simpler for long term perspective.

### Additional context

We'd discussed this RFC in fluent-bit issues:
https://github.com/fluent/fluent-bit/issues/6666

UPDATE:

Also, Fluent Bit with out_forward just uses `options` to represent their metadata. We don't expect any breakage for sending logs type of Fluent Bit events from Fluent Bit to Fluentd.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.