hashicorp / hashicorp/nomad

When running docker plugin with short lived jobs the stdout\stderr log files will occasionally be empty

Open
#6,931 9 comments 8 reactions 0 assignees View on GitHub
theme/driver/docker theme/logging type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
0.10.0-dev

### Operating system and Environment details
Debian 10 on AWS

### Issue
On very short runs, the dockerLogger fails to attach to docker's std streams, resulting with empty stdout \ stderr logs.

We noticed on our tests that when spinning jobs that are supposed to make very short operations and stop we occasionally get empty stdout \ stderr log files.

This behavior will not reproduce on 100% of the times we run, as far as we could understand the reason for the random behavior stems from the randomness of the function nextBackoff in file: https://github.com/hashicorp/nomad/blob/master/drivers/docker/docklog/docker_logger.go#L70 .
The dockerLogger will try to attach to the docker streams every backoff seconds.
Backoff is calculated as: " backoff * 1.15 * (1.0 + rand.Float64()) "

This also explains to the closed issue: https://github.com/hashicorp/nomad/issues/2457

Even tough a short lived script doesn't seem like a logical use case for Nomad, we successfully reproduced the issue with a python script that has a syntax error, in that case the stderr will be empty and we will not know why.

### Reproduction steps
create a docker image with a python script that has no requirements and has a very short execution time such as
`print("hello world")`
or with a syntax error
`print "hello world"`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with drivers/docker/docklog/docker_logger.go, especially the nextBackoff behavior and stream-attachment path described in the report. Reproduce the issue with a very short Docker job and the provided Python syntax-error example, then verify that stdout and stderr logs are populated reliably for short-lived jobs.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
devops, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.