Allocation with ephemeral_disk is garbage collected before migration to new node is completed
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
1.5.2
### Operating system and Environment details
Ubuntu 20.04
### Issue
Setup:
Nomad client node uses default setting `gc_disk_usage_threshold = 80`
Client node disk is used up to 85%
We have a group with ephemeral_disk:
```
group "example" {
ephemeral_disk {
migrate = true
sticky = true
}
}
```
after draining a node on which the job is running, it fails to reallocate disk to new node with `error making snapshot`
bacuse garbage collection manages to be faster than the reallocation.
Shouldn't the gc process wait until the reallocation is completed?
### Reproduction steps
- Fill disk higher than `gc_disk_usage_threshold`
- Craete job that uses ephemeral disk
- Drain node on which the job is running
#### Expected Result
`ephemeral_disk` is moved firtst, then the allocation is garbage collected
#### Actual Result
allocation is garbage collected before it manages to move the `ephemeral_disk`
### Job file (if appropriate)
### Nomad Server logs (if appropriate)
### Nomad Client logs (if appropriate)
```
2023-03-29T10:13:55.115+0200 [ERROR] http: request failed: method=GET path=/v1/client/allocation/ad482d74-9008-7ef0-3799-ca523dbf83fd/snapshot error="error making snapshot: failed to snapshot /opt/nomad/alloc/ad482d74-9008-7ef0-3799-ca523dbf83fd/alloc/data: lstat /opt/nomad/alloc/ad482d74-9008-7ef0-3799-ca523dbf83fd/alloc/data: no such file or directory" code=500
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the issue on Nomad 1.5.2 by exceeding gc_disk_usage_threshold, creating a job with migrate and sticky ephemeral_disk settings, and draining its node. Start with the allocation snapshot request at /v1/client/allocation/{id}/snapshot and the reported error logs. Done means the ephemeral disk migration completes before garbage collection removes the allocation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, ubuntu
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100