alloc fs doesn't have access to mounted volumes
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
The various Nomad filesystem commands (and their APIs) don't have access to the contents of mounted volumes. This means that while tasks can access host volume content (and CSI volume content), `nomad alloc fs` cannot. The operator can view the files with `nomad alloc exec`, but as this requires a higher level of privilege with its own ACL, this workaround isn't great.
Example jobspec that writes to a volume mount location in the task dir. (The exact location doesn't matter so long as its somewhere within the alloc dir.)
```hcl
# a job that mounts an EBS volume and writes its alloc ID as a file
job "use-ebs-volume" {
datacenters = ["dc1"]
group "group" {
volume "test" {
type = "csi"
source = "ebs-vol0"
}
task "task" {
driver = "docker"
config {
image = "busybox:1"
command = "/bin/sh"
args = ["-c", "touch /local/test/${NOMAD_ALLOC_ID}; sleep 3600"]
}
volume_mount {
volume = "test"
destination = "${NOMAD_TASK_DIR}/test"
read_only = false
}
resources {
cpu = 500
memory = 128
}
}
}
}
```
Results below. Note that this all works as expected when there's no `volume_mount` stanza.
```
▶ nomad alloc fs d7db81c8 task/local
Mode Size Modified Time Name
drwxr-xr-x 4.0 KiB 2020-03-17T15:22:46Z test/
▶ nomad alloc fs d7db81c8 task/local/test
Mode Size Modified Time Name
```
Exec into the allocation and see everything we'd expect to see:
```
▶ nomad alloc exec d7db81c8 /bin/sh
/ # df -h
Filesystem Size Used Available Use% Mounted on
overlay 7.7G 3.7G 4.0G 49% /
tmpfs 64.0M 0 64.0M 0% /dev
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
shm 64.0M 0 64.0M 0% /dev/shm
/dev/xvda1 7.7G 3.7G 4.0G 49% /local
tmpfs 1.0M 0 1.0M 0% /secrets
/dev/xvda1 7.7G 3.7G 4.0G 49% /alloc
/dev/xvdba 39.2G 48.0M 39.2G 0% /local/test
/dev/xvda1 7.7G 3.7G 4.0G 49% /etc/resolv.conf
/dev/xvda1 7.7G 3.7G 4.0G 49% /etc/hostname
/dev/xvda1 7.7G 3.7G 4.0G 49% /etc/hosts
tmpfs 1.9G 0 1.9G 0% /proc/acpi
tmpfs 64.0M 0 64.0M 0% /proc/kcore
tmpfs 64.0M 0 64.0M 0% /proc/keys
tmpfs 64.0M 0 64.0M 0% /proc/timer_list
tmpfs 64.0M 0 64.0M 0% /proc/timer_stats
tmpfs 64.0M 0 64.0M 0% /proc/sched_debug
tmpfs 1.9G 0 1.9G 0% /proc/scsi
tmpfs 1.9G 0 1.9G 0% /sys/firmware
/ # cd /local/test
/local/test # ls -lah
total 24K
drwxr-xr-x 3 root root 4.0K Mar 17 15:22 .
drwxrwxrwx 3 nobody nogroup 4.0K Mar 17 15:22 ..
-rw-r--r-- 1 root root 0 Mar 17 15:13 5564b554-893e-41e3-8d2b-62316c35dbf6
-rw-r--r-- 1 root root 0 Mar 17 15:07 5f54d08e-2fdf-9b4b-2f6b-4a7b52cb8e9f
-rw-r--r-- 1 root root 0 Mar 17 15:11 68ad6e66-5da4-04d2-0ede-ec9c96d6d204
-rw-r--r-- 1 root root 0 Mar 17 14:31 b7f43bb5-15a3-b8ee-ed44-0bc3156e8f78
-rw-r--r-- 1 root root 0 Mar 17 15:04 d56136d6-550b-c379-1d2f-a29207a3232d
-rw-r--r-- 1 root root 0 Mar 17 15:22 d7db81c8-4c4c-a10c-fa47-1e7ee2e5854d
-rw-r--r-- 1 root root 0 Mar 17 14:59 e7f39ca6-6df4-f4b8-13c3-61ca2c00c209
-rw-r--r-- 1 root root 0 Mar 17 15:15 ed9a2dd6-bbe7-9498-dfac-6a0dc1bf6ab0
drwx------ 2 root root 16.0K Mar 17 14:31 lost+found
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the `nomad alloc fs` command and the filesystem APIs, then reproduce the provided jobspec and compare their output with `nomad alloc exec`. Trace how mounted volume paths are handled within the allocation directory. Done means the filesystem commands and APIs can display contents from mounted host and CSI volumes while preserving existing behavior for ordinary allocation files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, backend, cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100