manager/csi/manager.go: getPlugin: Issue with handling plugin names related to image tags

Open
#3,215 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
go

Research direction

Start in manager/csi/manager.go at getPlugin and handleNode, comparing the plugin name from info.PluginName with v.Spec.Driver.Name. Trace how aliases and image tags create plugin instances, then verify that both startup and usage resolve to the same instance and that CSI volume publishing no longer reports a missing node ID.

Written by the indexing model from the issue text.

Description

I was working on debugging compatibility with the CephCSI driver and stumbled upon an issue in how plugin names and aliases are handled.

Installed the driver using this cmd:

docker plugin rm rbd.csi.ceph.com --force && \
docker plugin install <privaterepo>/ceph-csi-swarm/cephcsi-aio:canary \
--grant-all-permissions \
NODE_ID=test-docker-1 \
CEPHCSI_VERBOSITY=10 \
DEBUG_ENTRYPOINT=true \
--alias rbd.csi.ceph.com

Created CSI volume and a service using it fine. Basically, everything worked up until volume publishing, where I was met with an error message from Docker stating:
CSI node ID not found for given Swarm node ID.

Looking at the code:

That log only triggers if this lookup fails:

csiNodeID := p.swarmToCSI[nodeID]
if csiNodeID == "" { ... }

There's two maps in the plugin object swarmToCSI and csiToSwarm.
These are correctly populated upon daemon startup, which I checked using debug statements:

Image

At the point in time where these maps are accessed, they seem to be emtpy:

Image

I further tested this by logging the ptr of the underlying plugin object which confirmed there were two "virtual" instances of the same plugin at runtime:
At start:

Image

At use:

Image

Looking at where plugin object instances are created, manager/csi/manager.go, and the usages thereof, it seems like handleNode gets the plugin's name from info.PluginName, (https://github.com/moby/swarmkit/blob/3a23580599a45034467695b859a0e794e62bddfa/manager/csi/manager.go#L406) whereas all other codepaths get it from v.Spec.Driver.Name.

This effectively causes two driver objects to be created from one installed driver, one with the tag in the alias and one without, where one instance is correctly populated with values at startup, but the other unpopulated instance is what is actually used later in the code, causing the CSI driver to fail at NodePublishVolume.


To fix this I modified the getPlugin method to normalize/canonicalize plugin names correctly.
https://github.com/ppignet/swarmkit/tree/csi/manager_normalize_plugin_name

I confirmed that this fixes the issue by veryfing the ptr is now identical:

upon startup:
Image

upon usage:

Image Image

..., and now being able to spin up many volumes and services without errors anymore! 😺

Lmk if this is a non-issue and just me doing something wrong. But I think this addition would make handling more fail-proof in any case.

Dominant language
Go
Stars
3.7k
Forks
676
Avg merge
4d 9h
Merged PRs (30d)
6

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from moby/swarmkit

All issues in moby/swarmkit

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.