Nomad fails to restart on windows nodes with "funlock error: The segment is already unlocked" error
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
1.0.1
### Operating system and Environment details
Windows
### Issue
Nomad is unable to restart with following error
```
2021-02-24T17:09:51.874Z [WARN] agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=C:\ProgramData\nomad\plugins
2021-02-24T17:09:51.878Z [INFO] agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
2021-02-24T17:09:51.878Z [INFO] agent: detected plugin: name=exec type=driver plugin_version=0.1.0
2021-02-24T17:09:51.878Z [INFO] agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
2021-02-24T17:09:51.878Z [INFO] agent: detected plugin: name=java type=driver plugin_version=0.1.0
2021-02-24T17:09:51.878Z [INFO] agent: detected plugin: name=docker type=driver plugin_version=0.1.0
2021-02-24T17:09:51.878Z [INFO] client: using state directory: state_dir=C:\ProgramData\nomad\client
2021/02/24 17:09:56.896817 bolt.Close(): funlock error: The segment is already unlocked.
```
### Reproduction steps
There isn't a specific scenario that leads to this state but we have seen this happening on multiple clusters which are long running, specifically on the windows nodes. The nomad service on windows nodes goes in loop trying to restart and failing with the same error.
Removing the client state db file and restarting the nomad solves the issue but needs manual intervention and the issue can reoccur too.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the Windows restart loop and tracing the client state database lifecycle around the logged bolt.Close() error. The fix is complete when Nomad can restart on affected Windows nodes without deleting the client state database; add regression coverage if the relevant test location is identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100