hashicorp / hashicorp/nomad

artifact: Intermittent download skip on Windows with periodic jobs using crons and multiple tasks sharing same bucket (regression from #27398)

Open
#28,150 13 comments 0 reactions 0 assignees View on GitHub
hcc/jira theme/artifact type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

Nomad version
Nomad v2.0.3
Also reproduced on v1.11.2 and v1.11.3. Last known good version: v1.11.1.

Operating system and Environment details

OS: Windows Server
Driver: raw_exec
Job type: periodic batch (using crons scheduler)
Related issue / suspected regression source: #27398

Issue
Since Nomad 1.11.2, periodic jobs on Windows intermittently skip the artifact download phase entirely. The task starts directly without going through Received, Task Setup, or Downloading Artifacts events, causing it to execute with missing files and fail with exit code 1.
The regression is consistently reproduced when a job has two or more tasks each with their own artifact block, both downloading from the same S3/MinIO bucket, scheduled via crons. In those cases the client appears to skip the download phase on subsequent periodic invocations, likely because the artifact caching logic introduced in #27398 evaluates files left on disk from a prior run as already present.
The fix in #27398 (which resolved the renameat ... Access is denied issue on Windows) appears to have introduced this new behavior.

Reproduction steps

Run a periodic job using crons with the raw_exec driver on a Windows node.
Define two or more tasks, each with its own artifact block pointing to the same S3/MinIO bucket.
Let the job trigger multiple times via its cron schedule (typically fails within 3–5 runs).
Observe that some allocs skip the download phase entirely and start the task with missing files.

Expected Result
Every alloc should emit the full lifecycle event sequence:
Received → Task Setup → Downloading Artifacts → Started

Actual Result
Some allocs skip directly to Started with no Downloading Artifacts event, then fail because expected files are absent:
# Failing alloc — Downloading Artifacts event missing
Time Type Description
2026-06-17T19:51:54Z Not Restarting Exceeded allowed attempts 3
2026-06-17T19:51:54Z Terminated Exit Code: 1
2026-06-17T19:51:54Z Started Task started by client
2026-06-17T19:51:37Z Restarting Task restarting in 17.1s
2026-06-17T19:51:37Z Terminated Exit Code: 1
2026-06-17T19:51:17Z Started Task started by client
# Successful alloc — correct behavior
Time Type Description
2026-06-17T19:51:09Z Terminated Exit Code: 0
2026-06-17T19:50:48Z Started Task started by client
2026-06-17T19:50:46Z Downloading Artifacts Client is downloading artifacts
2026-06-17T19:50:45Z Task Setup Building Task Directory
2026-06-17T19:50:45Z Received Task received by client

Job file (if appropriate)
hcljob "example" {
datacenters = ["dc1"]
type = "batch"

periodic {
crons = ["*/5 * * * *"]
prohibit_overlap = true
}

group "grp" {

task "app" {
driver = "raw_exec"

artifact {
source = "s3::https://minio.example.com/bucket/app.zip"
destination = "local/"
options {
aws_access_key_id = "..."
aws_access_key_secret = "..."
}
}

artifact {
source = "s3::https://minio.example.com/bucket/config.yml"
destination = "local/app/etc"
options {
aws_access_key_id = "..."
aws_access_key_secret = "..."
archive = false
}
}

config {
command = "cmd.exe"
args = ["/c", "cd /d local\\app && app.exe -f config.yml"]
}
}

task "sidecar" {
driver = "raw_exec"

artifact {
source = "s3::https://minio.example.com/bucket/sidecar.zip"
destination = "local/"
options {
aws_access_key_id = "..."
aws_access_key_secret = "..."
}
}

artifact {
source = "s3::https://minio.example.com/bucket/sidecar-config.yml"
destination = "local/sidecar/etc"
options {
aws_access_key_id = "..."
aws_access_key_secret = "..."
archive = false
}
}

config {
command = "cmd.exe"
args = ["/c", "cd /d local\\sidecar && sidecar.exe -f sidecar-config.yml"]
}
}

}
}

Both tasks download from the same MinIO bucket (minio.example.com/bucket). The intermittent skip is observed on the second or subsequent cron-triggered runs when artifact files from a prior alloc remain on disk.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the client artifact-caching path introduced in #27398 and its interaction with periodic jobs using the crons scheduler, raw_exec, and multiple tasks. Reproduce on Windows Server with the provided job across several runs, then verify that each allocation emits Received, Task Setup, Downloading Artifacts, and Started events without missing files.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.