hashicorp / hashicorp/nomad

after client restart, failed task restore leads to cpuset error

Open
#16,626 2 comments 2 reactions 0 assignees View on GitHub
hcc/jira hcc/tfc theme/cgroups type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

While working on another issue, I had a task that failed to restore after restarting the client. When this happened, the cpuset manager started emitting an error once a second into the debug logs:

> 2023-03-23T16:32:15.442-0400 [DEBUG] client.driver_mgr.docker: failed to copy cpuset: driver=docker error="openat2 /sys/fs/cgroup/nomad.slice/d9a1cc4b-f066-b47d-7411-7963a6fa70ac.redis.scope/cpuset.cpus: no such file or directory"

The allocation ID here is the alloc ID of the previously existing allocation that failed to restore. It's replacement allocation has a scope directory as I'd expect.

Ref: https://hashicorp.atlassian.net/browse/NET-12137

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Docker driver and its cpuset manager, then reproduce the failed task-restore flow after a client restart. Trace the handling of the old allocation's missing cpuset.cpus path and compare it with the replacement allocation's scope directory. Done means the restore path no longer emits the repeated cpuset error.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.