When running docker plugin with short lived jobs the stdout\stderr log files will occasionally be empty
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
0.10.0-dev
### Operating system and Environment details
Debian 10 on AWS
### Issue
On very short runs, the dockerLogger fails to attach to docker's std streams, resulting with empty stdout \ stderr logs.
We noticed on our tests that when spinning jobs that are supposed to make very short operations and stop we occasionally get empty stdout \ stderr log files.
This behavior will not reproduce on 100% of the times we run, as far as we could understand the reason for the random behavior stems from the randomness of the function nextBackoff in file: https://github.com/hashicorp/nomad/blob/master/drivers/docker/docklog/docker_logger.go#L70 .
The dockerLogger will try to attach to the docker streams every backoff seconds.
Backoff is calculated as: " backoff * 1.15 * (1.0 + rand.Float64()) "
This also explains to the closed issue: https://github.com/hashicorp/nomad/issues/2457
Even tough a short lived script doesn't seem like a logical use case for Nomad, we successfully reproduced the issue with a python script that has a syntax error, in that case the stderr will be empty and we will not know why.
### Reproduction steps
create a docker image with a python script that has no requirements and has a very short execution time such as
`print("hello world")`
or with a syntax error
`print "hello world"`
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with drivers/docker/docklog/docker_logger.go, especially the nextBackoff behavior and stream-attachment path described in the report. Reproduce the issue with a very short Docker job and the provided Python syntax-error example, then verify that stdout and stderr logs are populated reliably for short-lived jobs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- devops, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100