hashicorp / hashicorp/nomad

alloc fs doesn't have access to mounted volumes

Open
#7,365 4 comments 0 reactions 0 assignees View on GitHub
theme/storage type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

The various Nomad filesystem commands (and their APIs) don't have access to the contents of mounted volumes. This means that while tasks can access host volume content (and CSI volume content), `nomad alloc fs` cannot. The operator can view the files with `nomad alloc exec`, but as this requires a higher level of privilege with its own ACL, this workaround isn't great.

Example jobspec that writes to a volume mount location in the task dir. (The exact location doesn't matter so long as its somewhere within the alloc dir.)

```hcl
# a job that mounts an EBS volume and writes its alloc ID as a file
job "use-ebs-volume" {
datacenters = ["dc1"]

group "group" {
volume "test" {
type = "csi"
source = "ebs-vol0"
}

task "task" {
driver = "docker"

config {
image = "busybox:1"
command = "/bin/sh"
args = ["-c", "touch /local/test/${NOMAD_ALLOC_ID}; sleep 3600"]
}

volume_mount {
volume = "test"
destination = "${NOMAD_TASK_DIR}/test"
read_only = false
}

resources {
cpu = 500
memory = 128
}
}
}
}

```

Results below. Note that this all works as expected when there's no `volume_mount` stanza.

```
▶ nomad alloc fs d7db81c8 task/local
Mode Size Modified Time Name
drwxr-xr-x 4.0 KiB 2020-03-17T15:22:46Z test/

▶ nomad alloc fs d7db81c8 task/local/test
Mode Size Modified Time Name
```

Exec into the allocation and see everything we'd expect to see:

```
▶ nomad alloc exec d7db81c8 /bin/sh
/ # df -h
Filesystem Size Used Available Use% Mounted on
overlay 7.7G 3.7G 4.0G 49% /
tmpfs 64.0M 0 64.0M 0% /dev
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
shm 64.0M 0 64.0M 0% /dev/shm
/dev/xvda1 7.7G 3.7G 4.0G 49% /local
tmpfs 1.0M 0 1.0M 0% /secrets
/dev/xvda1 7.7G 3.7G 4.0G 49% /alloc
/dev/xvdba 39.2G 48.0M 39.2G 0% /local/test
/dev/xvda1 7.7G 3.7G 4.0G 49% /etc/resolv.conf
/dev/xvda1 7.7G 3.7G 4.0G 49% /etc/hostname
/dev/xvda1 7.7G 3.7G 4.0G 49% /etc/hosts
tmpfs 1.9G 0 1.9G 0% /proc/acpi
tmpfs 64.0M 0 64.0M 0% /proc/kcore
tmpfs 64.0M 0 64.0M 0% /proc/keys
tmpfs 64.0M 0 64.0M 0% /proc/timer_list
tmpfs 64.0M 0 64.0M 0% /proc/timer_stats
tmpfs 64.0M 0 64.0M 0% /proc/sched_debug
tmpfs 1.9G 0 1.9G 0% /proc/scsi
tmpfs 1.9G 0 1.9G 0% /sys/firmware

/ # cd /local/test
/local/test # ls -lah
total 24K
drwxr-xr-x 3 root root 4.0K Mar 17 15:22 .
drwxrwxrwx 3 nobody nogroup 4.0K Mar 17 15:22 ..
-rw-r--r-- 1 root root 0 Mar 17 15:13 5564b554-893e-41e3-8d2b-62316c35dbf6
-rw-r--r-- 1 root root 0 Mar 17 15:07 5f54d08e-2fdf-9b4b-2f6b-4a7b52cb8e9f
-rw-r--r-- 1 root root 0 Mar 17 15:11 68ad6e66-5da4-04d2-0ede-ec9c96d6d204
-rw-r--r-- 1 root root 0 Mar 17 14:31 b7f43bb5-15a3-b8ee-ed44-0bc3156e8f78
-rw-r--r-- 1 root root 0 Mar 17 15:04 d56136d6-550b-c379-1d2f-a29207a3232d
-rw-r--r-- 1 root root 0 Mar 17 15:22 d7db81c8-4c4c-a10c-fa47-1e7ee2e5854d
-rw-r--r-- 1 root root 0 Mar 17 14:59 e7f39ca6-6df4-f4b8-13c3-61ca2c00c209
-rw-r--r-- 1 root root 0 Mar 17 15:15 ed9a2dd6-bbe7-9498-dfac-6a0dc1bf6ab0
drwx------ 2 root root 16.0K Mar 17 14:31 lost+found

```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the `nomad alloc fs` command and the filesystem APIs, then reproduce the provided jobspec and compare their output with `nomad alloc exec`. Trace how mounted volume paths are handled within the allocation directory. Done means the filesystem commands and APIs can display contents from mounted host and CSI volumes while preserving existing behavior for ordinary allocation files.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.