hashicorp / hashicorp/nomad

Snapshot failing on Windows

Open
#11,796 2 comments 0 reactions 0 assignees View on GitHub
hcc/cst stage/needs-investigation theme/platform-windows type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Output from `nomad version`

### Operating system and Environment details
OS Name: Microsoft Windows Server 2019 Standard (x64)
OS Version: 1809 Build 17763.2366

**Nomad Version**
Nomad v1.2.3

### Issue
When trying to do a snapshot on Windows it fails with:
`PS C:\> nomad.exe operator snapshot save backup.nomad.bak
Filed to finalize snapshot file: rename backup.nomad.bak.tmp backup.nomad.bak: The process cannot access the file because it is being used by another process.`

The log shows:
`2022-01-07T10:25:05.816+0100 [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:59692`
`2022-01-07T10:25:05.842+0100 [INFO] nomad.raft: starting snapshot up to: index=34932`
`2022-01-07T10:25:05.842+0100 [INFO] snapshot: creating new snapshot: path=C:\Nomad\Data\server\raft\snapshots\6-34932-1641547505842.tmp`
`2022-01-07T10:25:05.880+0100 [WARN] snapshot: found temporary snapshot: name=4-24581-1640983949398.tmp`
`2022-01-07T10:25:05.880+0100 [INFO] snapshot: reaping snapshot: path=C:\Nomad\Data\server\raft\snapshots\6-34928-1641547315471`
`2022-01-07T10:25:05.882+0100 [INFO] nomad.raft: no logs to truncate`
`2022-01-07T10:25:05.883+0100 [INFO] nomad.raft: snapshot complete up to: index=34932`
`2022-01-07T10:25:05.908+0100 [DEBUG] http: request complete: method=GET path=/v1/operator/snapshot duration=66.1992ms`

### Reproduction steps
Run `nomad.exe operator snapshot save backup.nomad.bak`

#### Expected Result
To have a working snapshot

#### Actual Result
The command fails, but still manages to create a file called `backup.nomad.bak.tmp`

I have tried disabling Windows Defender to see if that was locking the file, but that wasn´t the case.
It fails on all the Nomad servers and clients we have (3 servers, 2 clients). We are still in the very early adoption phase of Nomad.

We have a similar problem with Consul although a different error is thrown:
`consul.exe snapshot save backup.consul.bak`
`Error writing unverified snapshot file: sync .: The handle is invalid.`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing `nomad.exe operator snapshot save backup.nomad.bak` on Windows and trace the snapshot save operation through the temporary-file and rename steps described in the report. Done means the command creates a working `backup.nomad.bak` without leaving a failing `.tmp` file.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.