hashicorp / hashicorp/nomad

tmpfs mount leak

Open
#7,848 3 comments 0 reactions 0 assignees View on GitHub
theme/client type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

After a job is stopped, we appear to be leaking a mount for the tmpfs for the secrets dir. I noticed this during development work on `master` this morning, but I've confirmed it's an issue with the 0.11.x series, 0.10.x series, 0.9.7, and even 0.8.7.

```sh
$ nomad job status
ID Type Priority Status Submit Date
example service 50 running 2020-05-01T15:04:26Z

# see the mount
$ mount | grep redis
tmpfs on /var/nomad/alloc/71e15867-dc8f-7104-8625-e452169f2e73/redis/secrets type tmpfs (rw,noexec,relatime,size=1024k)

# stop the job
$ nomad job stop example
==> Monitoring evaluation "413c5179"
Evaluation triggered by job "example"
Evaluation within deployment: "b3db1a6d"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "413c5179" finished with status "complete"

# container is gone
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

# mount is still there
$ mount | grep redis
tmpfs on /var/nomad/alloc/71e15867-dc8f-7104-8625-e452169f2e73/redis/secrets type tmpfs (rw,noexec,relatime,size=1024k)

# what's in it?
$ sudo ls -lah /var/nomad/alloc/71e15867-dc8f-7104-8625-e452169f2e73/redis/secrets
total 4.0K
drwxrwxrwx 2 nobody nogroup 60 May 1 15:04 .
drwxrwxrwx 5 nobody nogroup 4.0K May 1 15:04 ..
-rw-r--r-- 1 root root 0 May 1 15:04 .nomad-mount

```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the leak with the shown `nomad job status`, `nomad job stop`, and `mount | grep redis` commands, then trace tmpfs secrets cleanup during allocation teardown. Done means stopping the job removes the secrets mount and leaves no residual `.nomad-mount` directory.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.