containers / containers/conmon

Provide logging behavior policies applied by conmon to stdout/stderr

Open
#84 18 comments 0 reactions 1 assignee Claimed by @giuseppe View on GitHub
Dominant language
C
Stars
499
Forks
150
Avg merge
1d 10h
Merged PRs (30d)
20

Description

Today `conmon` is engaging in a byte-capture of `stdout`/`stderr` pipes, recording data as a series of JSON documents in a log file. All bytes from both file descriptors are captured without discrimination. We also see that `conmon` tries to write to disk as fast as possible (but sequentially) with what it reads from both pipes.

There is a natural back-pressure presented by `conmon` to the processes in the container writing to `stdout`/`stderr` that results from the rate the disk can accepts `write()`s. It is usually the case that all `conmon` processes server containers write to the same disk. Which means it is possible for "noisy" containers to take up much of the available disk bandwidth leading to unwanted impacts of one container on another.

Further, the rate at which containers write to disk is independent of the rate at which log file readers (or scrapers; e.g. fluentd, fluent-bit, rsyslog, syslog-ng, filebeat, etc.) can read. Because log file rotation is performed independent of the readers and `conmon` on most platforms, it is possible for `conmon` to write more data than what a reader can take in before a log file is deleted behind the reader's back with it ever seeing it.

These two problems suggest that log behavior policies applied by `conmon` to the log stream will give administrators the chance to solve them.

There are three policy behaviors we can begin to consider, each applied to `stdout`/`stderr` independently:

1. `back-pressure`
- Given a rate specified in "bytes per interval", stop reading from the given pipe if the number of bytes read so far exceeds the limit for that interval, continuing to read again at the beginning of the next interval
- Suggested interval could just be 1 second by default, and/or allow it to be configured along with the # of bytes for that interval
2. `drop`
- Given a rate like specified for `back-pressure`, drop bytes read over the limit for the remainder of the interval, accepting them again at the start of the next interval
3. `ignore`
- Ignore all bytes read from a given pipe without writing them to disk.

This will allow an administrator to apply a policy to all `conmon` processes. Since `conmon` processes don't "know" about each other, coordination of the setting of that policy would have to be managed externally to `conmon`.

It is possible that `conmon` processes could periodically poll for configuration changes, or accept some sort of signal to re-evaluate the change.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.