On demand Docker log collection
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Proposal
I just took a look at the memory usage on my Nomad agents, and realized that the overhead of Docker log collection is crazy. On my small scale cluster (personal install with 4 nomad agents, 65 alloc running), it was using about 33% of the total used memory (mainly the nomad logmon and nomad docker_logger processes. I used the reported used memory by systemd, with and without disable_log_collection = true).
Disabling log collection (and using for example fluentd for the docker task driver) is a solution to this insane consumption, but we loose access to the container logs from the web interface or the nomad alloc logs cli, which is convenient for quickly debugging (faster than querying a central log aggregator).
Maybe one way to mitigate this would be to have a 3rd mode for log collection which would be on-demande : as soon as the log streaming API is called, the corresponding logmon & docker_logger process could be spawned (and which would be killed after some timeout)
### Use-cases
An on-demand log collection would suppress most of the memory overhead of log collection for the Docker driver, while still allowing logs to be displayed in the web interface or the nomad cli for ponctual debugging
### Attempted Solutions
Using an out of band log collector/aggregator and turning disable_log_collection globaly in Nomad's agent conf is a workarround, but loosing access to the logs from the web interface is a serious drawback.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing Nomad's Docker log collection and the log streaming API used by the web interface and alloc logs CLI; the issue names no specific files or tests. Determine the lifecycle and timeout requirements for on-demand logmon and docker_logger processes, with completion meaning container logs remain available on request without the usual continuous memory overhead.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- backend, cli, devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100