hashicorp / hashicorp/nomad

cpu oversubscription

Open
#23,920 4 comments 1 reaction 0 assignees View on GitHub
stage/needs-discussion theme/resource-utilization theme/scheduling type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Proposal
Please allow to opt-out of setting cpu-shares on the docker driver.

Nomad uses the "cpu" resources of a job config for planning/performing allocations and, in case of the docker driver, the value will then also be set as cpu shares (https://docs.docker.com/engine/containers/resource_constraints/#cpu) on starting a docker container. We're running nomad on our own hardware, so outside of general system performance considerations, we don't have to worry about consuming too many cpu cycles, contrarily to a paid cloud environment. As a form of low-effort, at your own risk cpu overprovisioning, it would be great if there was an option to tell nomad to simply not set any cpu shares setting when starting a docker container.

### Use-cases
We're running a cpu intensive service via nomad, that, even though it gets given most of the available cpu of a machine, still gets throttled to unusability.

### Attempted Solutions
Running the service via docker on a nomad client machine without nomad (i.e. starting it manually on the command line) and not setting cpu shares makes the service run fine. Ideally, we'd like to continue to run the service via nomad, so we can avoid running one service outside of nomad.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how the Docker driver converts a job's cpu resource into Docker cpu shares. Review the driver configuration and container-start path, then inspect existing resource-related tests if present. Done means an explicit opt-out can be configured and a Docker container starts without cpu shares while Nomad scheduling still uses the cpu resource.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.