ozontech / ozontech/file.d

Feature: Add file_multiline action to join Docker split container log chunks (~16KB)

Open
#982 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
504
Forks
263
Avg merge
4d 2h
Merged PRs (30d)
5

Description

Is your feature request related to a problem? Please describe.

When reading container logs via the file input plugin (Docker JSON log format, e.g. /var/log/containers/*.log), long log lines are split into multiple file records (~16KB per chunk). Each chunk becomes a separate event with a partial log field.

This breaks downstream processing: json_decode / decode fail on incomplete JSON, and fields like msg_body or logs_type are never parsed correctly.

The k8s input plugin already handles this via built-in k8s-multiline, but there is no equivalent for users who read the same files with the file input plugin.

Describe the solution you'd like

Add a new action plugin file_multiline that:

Joins sequential chunks from the same file stream into one event
Works on a configurable field (default: log)
Treats a chunk as complete when the field value ends with a real newline character \n
Supports split_event_size for very long messages (similar to k8s-multiline)
Is intended to run before json_decode / decode in the actions pipeline
Example:

actions:

  • type: file_multiline
    field: log
    split_event_size: 1000000
  • type: json_decode
    field: log
    Describe alternatives you've considered

join action with regex — not suitable: split chunks have no predictable start/continue pattern; content is arbitrary JSON/text inside log.

k8s input instead of file — works, but requires k8s-specific setup and metadata; not ideal when only file tailing is needed.

Pipeline event_timeout only — does not solve joining; only limits wait time for incomplete buffers.

Fixing only in application — avoids splitting at source, but does not help when logs are already written by Docker/CRI.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining the existing k8s-multiline implementation and the actions pipeline to understand event buffering and action configuration. Define how file_multiline tracks sequential chunks, newline completion, configurable fields, and split_event_size; it is done when Docker-split records are joined before json_decode or decode without affecting separate file streams.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.