Feature: Add file_multiline action to join Docker split container log chunks (~16KB)
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 504
- Forks
- 263
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 5
Description
Is your feature request related to a problem? Please describe.
When reading container logs via the file input plugin (Docker JSON log format, e.g. /var/log/containers/*.log), long log lines are split into multiple file records (~16KB per chunk). Each chunk becomes a separate event with a partial log field.
This breaks downstream processing: json_decode / decode fail on incomplete JSON, and fields like msg_body or logs_type are never parsed correctly.
The k8s input plugin already handles this via built-in k8s-multiline, but there is no equivalent for users who read the same files with the file input plugin.
Describe the solution you'd like
Add a new action plugin file_multiline that:
Joins sequential chunks from the same file stream into one event
Works on a configurable field (default: log)
Treats a chunk as complete when the field value ends with a real newline character \n
Supports split_event_size for very long messages (similar to k8s-multiline)
Is intended to run before json_decode / decode in the actions pipeline
Example:
actions:
- type: file_multiline
field: log
split_event_size: 1000000 - type: json_decode
field: log
Describe alternatives you've considered
join action with regex — not suitable: split chunks have no predictable start/continue pattern; content is arbitrary JSON/text inside log.
k8s input instead of file — works, but requires k8s-specific setup and metadata; not ideal when only file tailing is needed.
Pipeline event_timeout only — does not solve joining; only limits wait time for incomplete buffers.
Fixing only in application — avoids splitting at source, but does not help when logs are already written by Docker/CRI.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining the existing k8s-multiline implementation and the actions pipeline to understand event buffering and action configuration. Define how file_multiline tracks sequential chunks, newline completion, configurable fields, and split_event_size; it is done when Docker-split records are joined before json_decode or decode without affecting separate file streams.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100