logdna output includes the full log event in the line field
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 71
Description
## Bug Report
**Describe the bug**
The `out_logdna` plugin promotes primary keys (`meta`, `level`, `app`, `file`) from the record to top-level fields in the ingest payload, but then serializes the entire original record (including those same keys) as JSON into the `line` field. This causes field duplication, for example, `meta` appears both as a top-level object and inside the `line` JSON string.
On Mezmo's side, the duplicated `meta` inside `line` is rendered as `_meta` to avoid collision with the promoted top-level `meta`, resulting in users seeing both `meta` and `_meta` in the UI.
Per the [Mezmo ingest API spec](https://docs.mezmo.com/log-analysis-api#ingest), `line` is defined as "Text of the log line" — a plain text string, not a JSON dump of the full record.
**To Reproduce**
1. Configure an input that produces records with a `meta` field (e.g., via a Lua filter that sets `record["meta"] = {foo = "bar"}`)
2. Route to `out_logdna`
3. Observe the payload sent to Mezmo — `meta` appears both at the top level and inside the `line` JSON string
4. In Mezmo UI, the log line shows both `meta` and `_meta`
Without a logdna/Mezmo endpoint, the output can be redirected to a custom http server and capture the structure of the payload for validation.
config.yml
```yaml
service:
flush: 5
log_level: info
pipeline:
inputs:
- name: dummy
tag: test
rate: 1
filters:
- name: lua
match: test
call: add_meta
code: |
function add_meta(tag, timestamp, record)
record["meta"] = { source = "test", env = "dev" }
record["level"] = "info"
return 1, timestamp, record
end
outputs:
- name: logdna
match: test
api_key: ${LOGDNA_API_KEY}
hostname: test-host
app: test-app
```
**Expected behavior**
- Promoted keys (`meta`, `level`, `app`, `file`) should be excluded from the `line` field after being extracted
- Ideally, the plugin should look for a dedicated message key (e.g., `message`, `log`) to use as the `line` value, rather than serializing the entire record
**Your Environment**
* Version used: master (`commit 29deec9e`)
* Filters and plugins: `out_logdna`
**Additional context**
The extraction logic in `record_append_primary_keys()` correctly identifies and promotes primary keys. The issue is in `logdna_compose_payload()`, where `flb_msgpack_to_json_str()` serializes the full `log_event.body` without filtering out the already-promoted keys.
Contributor guide
Research direction
Start with record_append_primary_keys() and logdna_compose_payload(), which the issue identifies as the relevant entry points. Use the provided config with a custom HTTP server to capture the ingest payload, then verify that promoted keys are not duplicated in line and that the resulting line follows the ingest API's plain-text expectation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100