Task blocked on write to stdout
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v1.0.1 (c9c68aa55a7275f22d2338f2df53e67ebfcb9238)
### Operating system and Environment details
Linux 3.10.0-693.5.2.el7.x86_64
Using raw_exec driver
### Issue
Allocation that has been restarted due to template change, exited cleanly with 0, then template change restarted again, then exited with 138, due to SIGBUS 10 experiences an issue where the stdout named pipe is blocked. It appears logmon is no longer reading from the pipe. This halts all activity on the main thread since the write syscall is not able to complete.
See allocation Event log
```
Recent Events:
Time Type Description
2021-10-06T02:13:32-04:00 Started Task started by client
2021-10-06T02:13:32-04:00 Restarting Task restarting in 0s
2021-10-06T02:13:32-04:00 Terminated Exit Code: 138, Signal: 10
2021-10-06T02:13:32-04:00 Restart Signaled Template with change_mode restart re-rendered
2021-10-06T02:13:30-04:00 Started Task started by client
2021-10-06T02:13:30-04:00 Restarting Task restarting in 0s
2021-10-06T02:13:30-04:00 Terminated Exit Code: 0
2021-10-06T02:09:58-04:00 Restart Signaled Template with change_mode restart re-rendered
2021-10-05T18:03:29-04:00 Started Task started by client
2021-10-05T18:03:29-04:00 Task Setup Building Task Directory
```
### Reproduction steps
Can't directly reproduce.
#### Expected Result
Writes to stdout shouldn't block
#### Actual Result
Writes to stdout block
### Nomad Server logs (if appropriate)
I don't believe these are pertinent
### Nomad Client logs (if appropriate)
Never been able to find anything that looks apropos
Might be related to #7619 ? Not too sure.
BTW: It'd be nice if logmon had the allocation id in the command line. Hundreds of processes with the same command line that don't exist in any meaningful process hierarchy are annoying.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by investigating logmon and the raw_exec task lifecycle around template-triggered restarts, focusing on the stdout named pipe and the SIGBUS termination described in the allocation events. Check whether logmon stops reading after the restart, and use the reported expected result—stdout writes must not block—as the completion criterion. No direct reproduction or file and test locations are provided.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100