hashicorp / hashicorp/nomad

On demand Docker log collection

Open
#20,558 5 comments 2 reactions 0 assignees View on GitHub
theme/logging type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Proposal
I just took a look at the memory usage on my Nomad agents, and realized that the overhead of Docker log collection is crazy. On my small scale cluster (personal install with 4 nomad agents, 65 alloc running), it was using about 33% of the total used memory (mainly the nomad logmon and nomad docker_logger processes. I used the reported used memory by systemd, with and without disable_log_collection = true).

Disabling log collection (and using for example fluentd for the docker task driver) is a solution to this insane consumption, but we loose access to the container logs from the web interface or the nomad alloc logs cli, which is convenient for quickly debugging (faster than querying a central log aggregator).

Maybe one way to mitigate this would be to have a 3rd mode for log collection which would be on-demande : as soon as the log streaming API is called, the corresponding logmon & docker_logger process could be spawned (and which would be killed after some timeout)

### Use-cases
An on-demand log collection would suppress most of the memory overhead of log collection for the Docker driver, while still allowing logs to be displayed in the web interface or the nomad cli for ponctual debugging

### Attempted Solutions
Using an out of band log collector/aggregator and turning disable_log_collection globaly in Nomad's agent conf is a workarround, but loosing access to the logs from the web interface is a serious drawback.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing Nomad's Docker log collection and the log streaming API used by the web interface and alloc logs CLI; the issue names no specific files or tests. Determine the lifecycle and timeout requirements for on-demand logmon and docker_logger processes, with completion meaning container logs remain available on request without the usual continuous memory overhead.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
backend, cli, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.