hashicorp / hashicorp/nomad

Poor handling of "nomad volume status -verbose" for CSI plugins without LIST_VOLUMES capability

Open
#15,040 1 comment 0 reactions 0 assignees View on GitHub
stage/accepted theme/bad-ux theme/storage
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v1.3.3 (428b2cd8014c48ee9eae23f02712b7219da16d30)
Tested with this version but v1.4.1 code has the same issues (and is linked below).

### Operating system and Environment details
Red Hat Enterprise Linux release 8.4 (Ootpa)

### Issue
The [BeeGFS CSI Driver](https://github.com/NetApp/beegfs-csi-driver) does not support the [CSI LIST_VOLUMES capability](https://github.com/container-storage-interface/spec/blob/master/spec.md#listvolumes). When `nomad volume status -verbose` is run, a raw HTTP error code and the message "unimplemented for this plugin" are shown.

### Reproduction steps
- Deploy the BeeGFS CSI driver (or some other driver that doesn't support the LIST_VOLUMES capability) to a Nomad cluster.
- Run `nomad volume status -verbose`

#### Expected Result
Similar output to when `nomad volume status` is run, perhaps with an additional message indicating specifically that LIST_VOLUMES is not implemented or that listing external volumes is not supported by this plugin.

#### Actual Result
Similar output to when `nomad volume status` is run, followed by an raw HTTP error code and the message "unimplemented for this plugin".

### Nomad command output

```
webere@webere-dev:~/beegfs-csi-briver$ nomad volume status
Container Storage Interface
ID Name Plugin ID Schedulable Access Mode
beegfs-csi-volume beegfs-csi-volume beegfs-csi-plugin true multi-node-multi-writer

webere@webere-dev:~/beegfs-csi-briver$ nomad volume status --verbose
Container Storage Interface
ID Name Plugin ID Schedulable Access Mode
beegfs-csi-volume beegfs-csi-volume beegfs-csi-plugin true multi-node-multi-writer
Error querying CSI external volumes for plugin "beegfs-csi-plugin": Unexpected response code: 500 (rpc error: unimplemented for this plugin)
```

### Code deep dive

This only occurs because we execute "nomad volume status" with BOTH "-verbose" AND no specified ID.

No specified ID invokes "list mode":

https://github.com/hashicorp/nomad/blob/v1.4.1/command/volume_status_csi.go#L20-L23

No "-verbose" returns early:

https://github.com/hashicorp/nomad/blob/v1.4.1/command/volume_status_csi.go#L104-L106

Both conditions together ultimately result in the invocation of the ListVolumes RPC (and the documented failure message):

https://github.com/hashicorp/nomad/blob/v1.4.1/command/volume_status_csi.go#L128-L136

A Nomad server returns a 500 error code and the message "unimplemented for this plugin" when it recognizes that a driver doesn't advertise the LIST_VOLUMES capability.

https://github.com/hashicorp/nomad/blob/v1.4.1/nomad/csi_endpoint.go#L1189-L1191

We are supposed to see something like [this](https://developer.hashicorp.com/nomad/docs/commands/volume/status#examples) for drivers that support the LIST_VOLUMES capability. Nomad already quietly moves on without printing the external list portion if it thinks nothing went wrong and doesn't have a list of volumes.

https://github.com/hashicorp/nomad/blob/v1.4.1/command/volume_status_csi.go#L137-L140

It seems like we just need a way for Nomad to understand to move on quietly in this circumstance as well. Maybe the Nomad server should just return an empty list instead of a 500 error when the LIST_VOLUMES capability isn't supported? Maybe the Nomad command should know to look for "unimplemented for this plugin" in the error output and move on quietly (or optionally print something more informative)?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with command/volume_status_csi.go at the list-mode, verbose handling, and external-volume output paths, then inspect nomad/csi_endpoint.go where unsupported LIST_VOLUMES returns the error. Reproduce with a CSI plugin lacking LIST_VOLUMES; done means `nomad volume status --verbose` avoids the raw error while preserving normal volume output and clearly handles the unsupported capability.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.