hashicorp / hashicorp/nomad

Jobs aren't started ("Reattachment process not found") in single node Nomad

Open
#16,922 4 comments 0 reactions 0 assignees View on GitHub
stage/needs-investigation theme/platform-windows theme/plugin type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
1.4.2

### Operating system and Environment details
Windows

### Issue

HCL of the Nomad as a Windows service:
```
datacenter = "dc1"
data_dir = "C:\\ProgramData\\Foo\\Nomad\\data"
region = "de"
server {
enabled = true
bootstrap_expect = 1
}
client {
enabled = true
network_interface = "Ethernet"
}
plugin "raw_exec" {
config {
enabled = true
}
}
```
Sometimes the jobs are not started:
```
2023-04-12T09:41:11.180+0200 [ERROR] client.driver_mgr.raw_exec: failed to reattach to executor: driver=raw_exec error="error creating rpc client for executor plugin: Reattachment process not found" task_id=368c9cd0-a8c4-82b1-39c5-640fa6a6a04c/wf/d2176bd8
```

But sometimes the jobs are started eventually a few seconds later:
```
2023-04-12T09:39:41.569+0200 [DEBUG] client.driver_mgr.raw_exec.executor.stdio: received EOF, stopping recv loop: alloc_id=368c9cd0-a8c4-82b1-39c5-640fa6a6a04c driver=raw_exec task_name=wf err="rpc error: code = Unavailable desc = error reading from server: EOF"
2023-04-12T09:41:11.933+0200 [INFO] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:C:\Program Files\Foo\WorkflowHostCli.exe Args:[--Name A]}"
```

The whole behavior seems unpredictable. Did we miss something here?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the raw_exec driver's executor reattachment path, using the Windows service configuration and the "Reattachment process not found" log as the reproduction context. Trace why the executor becomes unavailable, then reproduce the delayed-start behavior on Nomad 1.4.2; done means the cause and a verified fix or clear configuration guidance are recorded.

Written by the indexing model from the issue text.

Assessment

Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.