Poor handling of "nomad volume status -verbose" for CSI plugins without LIST_VOLUMES capability
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v1.3.3 (428b2cd8014c48ee9eae23f02712b7219da16d30)
Tested with this version but v1.4.1 code has the same issues (and is linked below).
### Operating system and Environment details
Red Hat Enterprise Linux release 8.4 (Ootpa)
### Issue
The [BeeGFS CSI Driver](https://github.com/NetApp/beegfs-csi-driver) does not support the [CSI LIST_VOLUMES capability](https://github.com/container-storage-interface/spec/blob/master/spec.md#listvolumes). When `nomad volume status -verbose` is run, a raw HTTP error code and the message "unimplemented for this plugin" are shown.
### Reproduction steps
- Deploy the BeeGFS CSI driver (or some other driver that doesn't support the LIST_VOLUMES capability) to a Nomad cluster.
- Run `nomad volume status -verbose`
#### Expected Result
Similar output to when `nomad volume status` is run, perhaps with an additional message indicating specifically that LIST_VOLUMES is not implemented or that listing external volumes is not supported by this plugin.
#### Actual Result
Similar output to when `nomad volume status` is run, followed by an raw HTTP error code and the message "unimplemented for this plugin".
### Nomad command output
```
webere@webere-dev:~/beegfs-csi-briver$ nomad volume status
Container Storage Interface
ID Name Plugin ID Schedulable Access Mode
beegfs-csi-volume beegfs-csi-volume beegfs-csi-plugin true multi-node-multi-writer
webere@webere-dev:~/beegfs-csi-briver$ nomad volume status --verbose
Container Storage Interface
ID Name Plugin ID Schedulable Access Mode
beegfs-csi-volume beegfs-csi-volume beegfs-csi-plugin true multi-node-multi-writer
Error querying CSI external volumes for plugin "beegfs-csi-plugin": Unexpected response code: 500 (rpc error: unimplemented for this plugin)
```
### Code deep dive
This only occurs because we execute "nomad volume status" with BOTH "-verbose" AND no specified ID.
No specified ID invokes "list mode":
https://github.com/hashicorp/nomad/blob/v1.4.1/command/volume_status_csi.go#L20-L23
No "-verbose" returns early:
https://github.com/hashicorp/nomad/blob/v1.4.1/command/volume_status_csi.go#L104-L106
Both conditions together ultimately result in the invocation of the ListVolumes RPC (and the documented failure message):
https://github.com/hashicorp/nomad/blob/v1.4.1/command/volume_status_csi.go#L128-L136
A Nomad server returns a 500 error code and the message "unimplemented for this plugin" when it recognizes that a driver doesn't advertise the LIST_VOLUMES capability.
https://github.com/hashicorp/nomad/blob/v1.4.1/nomad/csi_endpoint.go#L1189-L1191
We are supposed to see something like [this](https://developer.hashicorp.com/nomad/docs/commands/volume/status#examples) for drivers that support the LIST_VOLUMES capability. Nomad already quietly moves on without printing the external list portion if it thinks nothing went wrong and doesn't have a list of volumes.
https://github.com/hashicorp/nomad/blob/v1.4.1/command/volume_status_csi.go#L137-L140
It seems like we just need a way for Nomad to understand to move on quietly in this circumstance as well. Maybe the Nomad server should just return an empty list instead of a 500 error when the LIST_VOLUMES capability isn't supported? Maybe the Nomad command should know to look for "unimplemented for this plugin" in the error output and move on quietly (or optionally print something more informative)?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with command/volume_status_csi.go at the list-mode, verbose handling, and external-volume output paths, then inspect nomad/csi_endpoint.go where unsupported LIST_VOLUMES returns the error. Reproduce with a CSI plugin lacking LIST_VOLUMES; done means `nomad volume status --verbose` avoids the raw error while preserving normal volume output and clearly handles the unsupported capability.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100