fluent / fluent/fluent-bit

Implement processor for journald to opentelemetry semantic mapping

Open
#12,383 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

**Is your feature request related to a problem? Please describe.**
Journald implements structured fields with certain semantic. Opentelemetry has their semantic convention with similar fields.

https://github.com/open-telemetry/opentelemetry-specification/pull/4995 tries to describe the mapping between these in the opentelemetry appendix.

**Describe the solution you'd like**

I would like to propose a built in processor that would convert from a jorunald input (systemd) to an opentelemetry spec compliant envelope with the semantic of the fields preserved.

**Describe alternatives you've considered**

- I tried to implement the mapping by combining the existing processorts (rename, etc.) but they don't allow all the required operataions-
- I also tried to implement this as lua and WASM plugin. This works but it adds additional dependencies to fluentbit.

- Last I also implemented it as a processor in C, this works, see https://github.com/bachp/fluent-bit/tree/journald-mapping

**Additional context**
- Currently I have an out of tree version of this plugin that I can keep using if this is not desired by upstream.
- **The main question to answer is if this processor is something that should live upstream in fluent-bit or external.**

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the out-of-tree C implementation at https://github.com/bachp/fluent-bit/tree/journald-mapping and the OpenTelemetry specification PR #4995 to understand the proposed field mapping. The issue is done when maintainers decide whether this belongs upstream or externally; if accepted, the processor should preserve journald semantics in an OpenTelemetry-compliant envelope.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, linux
Domain
observability-sre, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.