fluent / fluent/fluentd

Support to process metadata in logs type of events that defined in forward protocol v1.5

Open
#5,369 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

### Is your feature request related to a problem? Please describe.

Currently, Fluentd is only able to process ingesting logs type of events with the following grammar I guess:

```
Connection ::= <>*

Request ::= Message | Forward | PackedForward | nil

Message ::= [ Tag, Time, Record, Option? ]

Forward ::= [ Tag, MultiEventStream, Option? ]

MultiEventStream ::= [ Event* ]

PackedForward ::= [ Tag, MessagePackEventStream, Option? ]

MessagePackEventStream ::= <>*

Metadata ::= integer | EventTime

Event ::= [ Metadata, Record ]

Tag ::= string

Record ::= object

Option ::= object
```

But I Forward Protocol v1.5 defines metadata which is inside of the events like:

```
Metadata ::= integer | EventTime | [ EventTime, Hash* ]

Event ::= [ Metadata, Record ]
```

So, the supported grammar of ingesting records with Forward protocol in Fluentd should be:

```
Connection ::= <>*

Request ::= Message | Forward | PackedForward | nil

Message ::= [ Tag, Time, Record, Option? ]

Forward ::= [ Tag, MultiEventStream, Option? ]

MultiEventStream ::= [ Event* ]

PackedForward ::= [ Tag, MessagePackEventStream, Option? ]

MessagePackEventStream ::= <>*

Metadata ::= integer | EventTime | [ EventTime, Hash* ] # We need to support the 3rd part of metadata BNF

Event ::= [ Metadata, Record ]

Tag ::= string

Record ::= object

Option ::= object
```

This is because Fluent Bit core developers decided to send metadata inside of logs type of events by default.
So, we need to improve interoperability between Fluentd and Fluent Bit.

### Describe the solution you'd like

Ideally, sending with metadata should work between Fluentd and Fluent Bit.

### Describe alternatives you've considered

Currently, just turning off sending metadata that is inside of events with `retain_metadata_in_forward_mode=false` is able to send logs type of events from Fluent Bit to Fluentd.

### Additional context

This is really needed feature because just turning off sending metadata losing the metadata information which is related to OpenTelemetry attributes in Fluent Bit because we're handling them within our mechanism of metadata handling.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.