hashicorp / hashicorp/nomad

CSI-related empty directories left behind after client alloc GC

Open
#20,544 1 comment 0 reactions 0 assignees View on GitHub
theme/storage type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

When an allocation that mounts a CSI volume starts, we create directories under `$datadir/client/csi/node/$plugin_id` to stage and publish the mount point for the volume. This results in a directory tree like the following (where `55113309` consumes the volume and `13bf0a2a` and `60e62cfc` are plugins).

```sh
$ sudo tree /var/nomad/data/client/csi
/var/nomad/data/client/csi
├── controller
│   └── org.democratic-csi.nfs
├── node
│   └── org.democratic-csi.nfs
│   ├── per-alloc
│   │   └── 55113309-8135-fd80-112b-d9f0f2c4cc6f
│   │   └── csi-volume-nfs
│   │   └── rw-file-system-single-node-writer # <- this is a bind-mount mount point
│   │   └── test.txt
│   └── staging
│   └── csi-volume-nfs
│   └── rw-file-system-single-node-writer # <- this is a bind-mount mount point
│   └── test.txt
└── plugins
├── 13bf0a2a-7866-7ded-8436-2c53f1268a41
│   └── csi.sock
└── 60e62cfc-0bbd-19ff-8f4d-a97b8e17d5cd
└── csi.sock

14 directories, 4 files
```

When the allocation is stopped, we unmount the mount points but the per-alloc directory and staging directory for the volume is left behind in the CSI node plugin's working directory. That's would be ok if we cleaned it up during client GC (we do the same for the alloc dir, after all). But we don't:

```sh
$ sudo tree /var/nomad/data/client/csi
/var/nomad/data/client/csi
├── controller
│   └── org.democratic-csi.nfs
├── node
│   └── org.democratic-csi.nfs
│   ├── per-alloc
│   │   └── 55113309-8135-fd80-112b-d9f0f2c4cc6f # <-- this should be gone
│   │   └── csi-volume-nfs
│   └── staging
│   └── csi-volume-nfs # <-- this should be gone
└── plugins
├── 13bf0a2a-7866-7ded-8436-2c53f1268a41
│   └── csi.sock
└── 60e62cfc-0bbd-19ff-8f4d-a97b8e17d5cd
└── csi.sock

12 directories, 2 files
```

The "leak" here is a couple of empty directories, but we should tidy up after ourselves.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing CSI client allocation cleanup and client GC, comparing the existing allocation-directory cleanup with the CSI node plugin's per-alloc and staging paths described in the issue. Done means stopped allocations leave neither the per-alloc directory nor the volume staging directory behind, while plugin directories remain intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.