hashicorp / hashicorp/nomad

Allocation with ephemeral_disk is garbage collected before migration to new node is completed

Open
#16,708 1 comment 2 reactions 0 assignees View on GitHub
theme/client theme/data migration theme/drain type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
1.5.2

### Operating system and Environment details
Ubuntu 20.04

### Issue
Setup:
Nomad client node uses default setting `gc_disk_usage_threshold = 80`
Client node disk is used up to 85%

We have a group with ephemeral_disk:
```
group "example" {
ephemeral_disk {
migrate = true
sticky = true
}
}
```

after draining a node on which the job is running, it fails to reallocate disk to new node with `error making snapshot`
bacuse garbage collection manages to be faster than the reallocation.
Shouldn't the gc process wait until the reallocation is completed?

### Reproduction steps
- Fill disk higher than `gc_disk_usage_threshold`
- Craete job that uses ephemeral disk
- Drain node on which the job is running

#### Expected Result
`ephemeral_disk` is moved firtst, then the allocation is garbage collected

#### Actual Result
allocation is garbage collected before it manages to move the `ephemeral_disk`

### Job file (if appropriate)

### Nomad Server logs (if appropriate)

### Nomad Client logs (if appropriate)
```
2023-03-29T10:13:55.115+0200 [ERROR] http: request failed: method=GET path=/v1/client/allocation/ad482d74-9008-7ef0-3799-ca523dbf83fd/snapshot error="error making snapshot: failed to snapshot /opt/nomad/alloc/ad482d74-9008-7ef0-3799-ca523dbf83fd/alloc/data: lstat /opt/nomad/alloc/ad482d74-9008-7ef0-3799-ca523dbf83fd/alloc/data: no such file or directory" code=500
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the issue on Nomad 1.5.2 by exceeding gc_disk_usage_threshold, creating a job with migrate and sticky ephemeral_disk settings, and draining its node. Start with the allocation snapshot request at /v1/client/allocation/{id}/snapshot and the reported error logs. Done means the ephemeral disk migration completes before garbage collection removes the allocation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, ubuntu
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.