hashicorp / hashicorp/nomad

nomad alloc logs -f not stopping after exit 0 ?

Open
#26,693 2 comments 0 reactions 0 assignees View on GitHub
hcc/jira stage/accepted theme/allocation API theme/cli type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Question
clarification: what is expected behavior for `nomad alloc -f`

1) should `nomad alloc -f $ALLOC` close after task terminated ?
2) or should the 'stream' stay open indefinitely, in which case, perhaps from a 'user' perspective

if 1) please handle this as bug report.
if 2) no need to read further ... although when there is, example, exit 1, the streaming stops, so it seems inconsistent, but then I guess I will just work with `timeout $NR nomad alloc -f` to avoid long runtimes in some cases.

### Nomad version
Output from `nomad version`
```
Nomad v1.10.4
BuildDate 2025-08-12T20:48:32Z
Revision 62b195aaa535b2159d215eaf89e6f4a455d6f686

Nomad v1.9.7
BuildDate 2025-03-11T09:07:15Z
Revision f8695974efdbaa9f80ace06f0eec5bd4d2501035+CHANGES
```

### Operating system and Environment details

`Ubuntu 22.04.5`

### Issue
`nomad alloc logs -f $ALLOC` seems to stay 'alive' tailing logs, even if the allocation terminated with exit 0.

### Reproduction steps
```
cat << EOF > nomad-hello-world.hcl
job "helloworld" {
datacenters = ["*"]
type = "batch"
group "example" {
task "alpine" {
driver = "docker"
config {
image = "alpine"
args = [
"echo",
"hello_world",
]
}
}
}
}
EOF

nomad job plan nomad-hello-world.hcl
nomad job run -check-index 0 nomad-hello-world.hcl

export MAX_TRIES=20 TRY=0 ALLOC_ID=""

#loop until alloc done downloading image.
while [[ -z "$ALLOC_ID" && $TRY -lt $MAX_TRIES ]]; do
ALLOC_ID=$(nomad job status helloworld | awk '/run.*(running|complete).*ago/ {getline; print $1}')
if [[ -z "$ALLOC_ID" ]]; then
sleep 2
((TRY++))
echo "Waiting... ($TRY/$MAX_TRIES)"
fi
done

#show alloc status -short
nomad alloc status -short $ALLOC_ID

#echo hello_world should be quick, timeout with 60 seconds.
timeout 60 nomad alloc logs -f $ALLOC_ID

ALLOC_EXITCODE=$?
if test $ALLOC_EXITCODE -eq 0
then
echo "ALLOC_EXITCODE:$ALLOC_EXITCODE = OK -> PASS .. logs -f terminated properly"
else
nomad alloc status $ALLOC_ID
echo "ERROR ALLOC_EXITCODE=$ALLOC_EXITCODE timeout reached, logs -f was not closing after task dead/terminated/exit (0)"
fi
```

#### Expected Result
to get `ALLOC_EXITCODE:0 = OK -> PASS .. logs -f terminated properly`

#### Actual Result
`ERROR ALLOC_EXITCODE=124 timeout reached, logs -f was not closing after task dead/terminated/exit (0)`

```
+ Job: "helloworld"
+ Task Group: "example" (1 create)
+ Task: "alpine" (forces create)

Scheduler dry-run:
- All tasks successfully allocated.

Job Modify Index: 0
To submit the job with version verification run:

nomad job run -check-index 0 nomad-hello-world.hcl

When running the job with the check-index flag, the job will only be run if the
job modify index given matches the server-side version. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.

==> View this job in the Web UI: https://nomad-stagingmyinternal-domain.com/ui/jobs/helloworld@default

==> 2025-09-04T09:25:59+08:00: Monitoring evaluation "97ec7bc6"
2025-09-04T09:25:59+08:00: Evaluation triggered by job "helloworld"
2025-09-04T09:26:00+08:00: Allocation "5c78ab54" created: node "03e2e1fc", group "example"
2025-09-04T09:26:00+08:00: Evaluation status changed: "pending" -> "complete"
==> 2025-09-04T09:26:00+08:00: Evaluation "97ec7bc6" finished with status "complete"

==> View job details and metrics in the Web UI: https://nomad-stagingmyinternal-domain.com/ui/jobs/helloworld@default
Error querying allocation: Unexpected response code: 500 (alloc lookup failed: index error: Invalid UUID: encoding/hex: invalid byte: U+0068 'h')
Waiting... (1/20)

==> View job details and metrics in the Web UI: https://nomad-staging.myinternal-domain.com/ui/jobs/helloworld@default
ID = 5c78ab54-40d2-8191-3e3e-b5d71c736719
Name = helloworld.example[0]
Created = 4s ago
Modified = 2s ago

Tasks
Name State Last Event Time Lifecycle
alpine dead Terminated 2025-09-04T09:26:00+08:00 main

==> View allocation details in the Web UI: https://nomad-staging.myinternal-domain.com/ui/allocations/5c78ab54-40d2-8191-3e3e-b5d71c736719
hello_world
ID = 5c78ab54-40d2-8191-3e3e-b5d71c736719
Eval ID = 97ec7bc6
Name = helloworld.example[0]
Node ID = 03e2e1fc
Node Name = nomad-staging-node1
Job ID = helloworld
Job Version = 0
Client Status = complete
Client Description = All tasks have completed
Desired Status = run
Desired Description =
Created = 1m4s ago
Modified = 1m2s ago

Task "alpine" is "dead"
Task Resources:
CPU Memory Disk Addresses
100 MHz 300 MiB 300 MiB

Task Events:
Started At = 2025-09-04T01:26:00Z
Finished At = 2025-09-04T01:26:00Z
Total Restarts = 0
Last Restart = N/A

Recent Events:
Time Type Description
2025-09-04T09:26:00+08:00 Terminated Exit Code: 0
2025-09-04T09:26:00+08:00 Started Task started by client
2025-09-04T09:25:59+08:00 Driver Downloading image
2025-09-04T09:25:59+08:00 Task Setup Building Task Directory
2025-09-04T09:25:59+08:00 Received Task received by client

==> View allocation details in the Web UI: https://nomad-staging.myinternal-domain.com/ui/allocations/5c78ab54-40d2-8191-3e3e-b5d71c736719
ERROR ALLOC_EXITCODE=124 timeout reached, logs -f was not closing after task dead/terminated/exit (0)
```

### Job file (if appropriate)
```
job "helloworld" {
datacenters = ["*"]
type = "batch"
group "example" {
task "alpine" {
driver = "docker"
config {
image = "alpine"
args = [
"echo",
"hello_world",
]
}
}
}
}

```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the `nomad alloc logs -f` command and reproduce the provided Alpine batch job on Ubuntu using the supplied shell script. Compare behavior for exit code 0 with the reported exit-1 case, then verify that the command exits successfully after the completed allocation while still streaming logs as intended.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, docker, go
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.