fluent / fluent/fluent-bit

logdna output includes the full log event in the line field

Open
#11,754 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

## Bug Report

**Describe the bug**

The `out_logdna` plugin promotes primary keys (`meta`, `level`, `app`, `file`) from the record to top-level fields in the ingest payload, but then serializes the entire original record (including those same keys) as JSON into the `line` field. This causes field duplication, for example, `meta` appears both as a top-level object and inside the `line` JSON string.

On Mezmo's side, the duplicated `meta` inside `line` is rendered as `_meta` to avoid collision with the promoted top-level `meta`, resulting in users seeing both `meta` and `_meta` in the UI.

Per the [Mezmo ingest API spec](https://docs.mezmo.com/log-analysis-api#ingest), `line` is defined as "Text of the log line" — a plain text string, not a JSON dump of the full record.

**To Reproduce**

1. Configure an input that produces records with a `meta` field (e.g., via a Lua filter that sets `record["meta"] = {foo = "bar"}`)
2. Route to `out_logdna`
3. Observe the payload sent to Mezmo — `meta` appears both at the top level and inside the `line` JSON string
4. In Mezmo UI, the log line shows both `meta` and `_meta`

Without a logdna/Mezmo endpoint, the output can be redirected to a custom http server and capture the structure of the payload for validation.

config.yml

```yaml
service:
flush: 5
log_level: info

pipeline:
inputs:
- name: dummy
tag: test
rate: 1

filters:
- name: lua
match: test
call: add_meta
code: |
function add_meta(tag, timestamp, record)
record["meta"] = { source = "test", env = "dev" }
record["level"] = "info"
return 1, timestamp, record
end

outputs:
- name: logdna
match: test
api_key: ${LOGDNA_API_KEY}
hostname: test-host
app: test-app
```

**Expected behavior**

- Promoted keys (`meta`, `level`, `app`, `file`) should be excluded from the `line` field after being extracted
- Ideally, the plugin should look for a dedicated message key (e.g., `message`, `log`) to use as the `line` value, rather than serializing the entire record

**Your Environment**

* Version used: master (`commit 29deec9e`)
* Filters and plugins: `out_logdna`

**Additional context**

The extraction logic in `record_append_primary_keys()` correctly identifies and promotes primary keys. The issue is in `logdna_compose_payload()`, where `flb_msgpack_to_json_str()` serializes the full `log_event.body` without filtering out the already-promoted keys.

Contributor guide

Open the contributing guide

Research direction

Start with record_append_primary_keys() and logdna_compose_payload(), which the issue identifies as the relevant entry points. Use the provided config with a custom HTTP server to capture the ingest payload, then verify that promoted keys are not duplicated in line and that the resulting line follows the ingest API's plain-text expectation.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.