hashicorp / hashicorp/nomad

[RFC] Model priviliged execution environments in the task-driver API

Open
#6,554 9 comments 4 reactions 0 assignees View on GitHub
stage/needs-discussion theme/auth theme/driver type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

Note: This issue mostly contains initial background thoughts to prompt discussion and is not yet a well defined proposal.

### Background

Currently, Nomads permission model around _runtime_ permissions of a job exist only within the implementation of a driver. This means that we do not include them in our ACL system or take whether the features are enabled on a particular client into account in the scheduler.

This is fine if a Nomad cluster is a uniform fleet, but that is rarely the case in larger clusters, and currently requires users to add additional metadata to privileged clients, and constraints to jobs that require them. It also then allows _anyone_ with access to the `submit-job` permission in any namespace to get privileged access to those hosts.

As part of #5378 however, access to privileged containers will become more normal as CSI Plugins require privileged containers in order to be able to create new mounts in the system for publishing volumes. Although we operate on a _trusted operator_ security model, there are many valid cases where CSI plugins may want to be deployed, without granting trivial docker privilege escalation to all.

### Proposal

I'm proposing introducing a Nomad-level API for modelling process isolation levels. This change would introduce a `privileged` option at the `task` level in a Nomad Job, that would signal to Nomad that the job my only be placed on nodes where that driver exposes the `Privileged` execution capability. It would also allow the introduction of a `privileged-execution` capability to the ACL system.

#### Task Configuration

```hcl
task "foo" {
driver = "docker"
privileged = true
}
```

#### Driver API

This is currently mostly undefined, but it would mostly involve updating the `DriverCapabilities` to introduce a new field that plugins may opt in to, and introducing a privileged option to TaskConfig.

#### Opting nodes into privileged execution

Currently, Nomad requires you to configure drivers with support for privileged execution modes in a client configuration. After this change, you'll still be required to enable support on an individual client, but by default will require using the new configuration for privileged execution modes.

To allow for backwards compatibility and a cleaner upgrade path, we will also offer an option in driver configuration to retain the existing behavior for using privileged execution environments.

##### Docker

###### Example Config

```hcl
plugin "docker" {
config {
allow_privileged = true // existing field
legacy_privileged_behavior = true // allow the existing behavior to be used, default: false
}
}
```

##### Raw Exec

The Raw Exec driver will begin exposing an `Unconstrained` isolation capability when `legacy_privileged_behavior` is `false` which will require that a user has access to `privileged` execution modes.

```hcl
plugin "docker" {
config {
enabled = true // existing field
legacy_privileged_behavior = true // allow the existing behavior to be used, default: false
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Read the task-driver API definitions around DriverCapabilities and TaskConfig, then trace how the Docker and Raw Exec drivers expose privileged execution and how ACL permissions are modeled. Before implementation, turn the RFC into a decided proposal covering scheduler placement, client configuration, backwards compatibility, and acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
api, backend, infrastructure, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.