hashicorp / hashicorp/nomad

Nomad fails to restart on windows nodes with "funlock error: The segment is already unlocked" error

Open
#10,086 10 comments 0 reactions 0 assignees View on GitHub
hcc/cst stage/accepted theme/dependencies theme/platform-windows type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
1.0.1

### Operating system and Environment details
Windows

### Issue
Nomad is unable to restart with following error

```
2021-02-24T17:09:51.874Z [WARN] agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=C:\ProgramData\nomad\plugins
2021-02-24T17:09:51.878Z [INFO] agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
2021-02-24T17:09:51.878Z [INFO] agent: detected plugin: name=exec type=driver plugin_version=0.1.0
2021-02-24T17:09:51.878Z [INFO] agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
2021-02-24T17:09:51.878Z [INFO] agent: detected plugin: name=java type=driver plugin_version=0.1.0
2021-02-24T17:09:51.878Z [INFO] agent: detected plugin: name=docker type=driver plugin_version=0.1.0
2021-02-24T17:09:51.878Z [INFO] client: using state directory: state_dir=C:\ProgramData\nomad\client
2021/02/24 17:09:56.896817 bolt.Close(): funlock error: The segment is already unlocked.
```

### Reproduction steps
There isn't a specific scenario that leads to this state but we have seen this happening on multiple clusters which are long running, specifically on the windows nodes. The nomad service on windows nodes goes in loop trying to restart and failing with the same error.

Removing the client state db file and restarting the nomad solves the issue but needs manual intervention and the issue can reoccur too.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the Windows restart loop and tracing the client state database lifecycle around the logged bolt.Close() error. The fix is complete when Nomad can restart on affected Windows nodes without deleting the client state database; add regression coverage if the relevant test location is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.