hashicorp / hashicorp/nomad

Task blocked on write to stdout

Open
#11,278 4 comments 0 reactions 0 assignees View on GitHub
theme/logging type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version

Nomad v1.0.1 (c9c68aa55a7275f22d2338f2df53e67ebfcb9238)

### Operating system and Environment details

Linux 3.10.0-693.5.2.el7.x86_64

Using raw_exec driver

### Issue

Allocation that has been restarted due to template change, exited cleanly with 0, then template change restarted again, then exited with 138, due to SIGBUS 10 experiences an issue where the stdout named pipe is blocked. It appears logmon is no longer reading from the pipe. This halts all activity on the main thread since the write syscall is not able to complete.

See allocation Event log
```
Recent Events:
Time Type Description
2021-10-06T02:13:32-04:00 Started Task started by client
2021-10-06T02:13:32-04:00 Restarting Task restarting in 0s
2021-10-06T02:13:32-04:00 Terminated Exit Code: 138, Signal: 10
2021-10-06T02:13:32-04:00 Restart Signaled Template with change_mode restart re-rendered
2021-10-06T02:13:30-04:00 Started Task started by client
2021-10-06T02:13:30-04:00 Restarting Task restarting in 0s
2021-10-06T02:13:30-04:00 Terminated Exit Code: 0
2021-10-06T02:09:58-04:00 Restart Signaled Template with change_mode restart re-rendered
2021-10-05T18:03:29-04:00 Started Task started by client
2021-10-05T18:03:29-04:00 Task Setup Building Task Directory
```

### Reproduction steps

Can't directly reproduce.

#### Expected Result

Writes to stdout shouldn't block

#### Actual Result

Writes to stdout block

### Nomad Server logs (if appropriate)

I don't believe these are pertinent

### Nomad Client logs (if appropriate)

Never been able to find anything that looks apropos

Might be related to #7619 ? Not too sure.

BTW: It'd be nice if logmon had the allocation id in the command line. Hundreds of processes with the same command line that don't exist in any meaningful process hierarchy are annoying.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by investigating logmon and the raw_exec task lifecycle around template-triggered restarts, focusing on the stdout named pipe and the SIGBUS termination described in the allocation events. Check whether logmon stops reading after the restart, and use the reported expected result—stdout writes must not block—as the completion criterion. No direct reproduction or file and test locations are provided.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.