## `ceph-csi-operator` falls back to its hardcoded `cephcsi:v3.17.0` default instead of the rook-ceph chart's `v3.17.1`, because the `ceph-csi-drivers` chart never sets `operatorConfig.driverSpecDefaults.imageSet.name`
- Dominant language
- Go
- Stars
- 63
- Forks
- 59
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 19
Description
## `ceph-csi-operator` falls back to its hardcoded `cephcsi:v3.17.0` default instead of the rook-ceph chart's `v3.17.1`, because the `ceph-csi-drivers` chart never sets `operatorConfig.driverSpecDefaults.imageSet.name`
### Summary
On a fresh Rook v1.20.7 install (using the CSI-Operator path: `rook-ceph` chart + `ceph-csi-drivers` chart v1.0.4, both installed via `helm upgrade --install` with no custom `csi.*` overrides beyond the chart defaults), the `csi-cephfsplugin` / `csi-rbdplugin` node-plugin containers end up pinned to `quay.io/cephcsi/cephcsi:v3.17.0`, while the `rook-ceph` chart's own default (`csi.cephcsi.tag`) is `v3.17.1`. In an airgapped/offline environment where only the images referenced by the Helm chart defaults were pre-pulled (i.e. `v3.17.1`), this produces a permanent `ImagePullBackOff` on every node.
### Root cause
`internal/controller/driver_controller.go` in `ceph-csi-operator` builds its working image map like this:
```go
r.images = maps.Clone(imageDefaults) // internal/controller/defaults.go
imageSetSpec := opConfig.Spec.DriverSpecDefaults.ImageSet
if imageSetSpec != nil && imageSetSpec.Name != "" {
// load ConfigMap and merge over defaults
maps.Copy(r.images, imageSetCM.Data)
}
imageSetSpec := r.driver.Spec.ImageSet
if imageSetSpec != nil && imageSetSpec.Name != "" {
maps.Copy(r.images, imageSetCM.Data)
}
```
`internal/controller/defaults.go` (still true on `main` as of writing) hardcodes:
```go
var imageDefaults = map[string]string{
"provisioner": "registry.k8s.io/sig-storage/csi-provisioner:v6.2.0",
"attacher": "registry.k8s.io/sig-storage/csi-attacher:v4.12.0",
"resizer": "registry.k8s.io/sig-storage/csi-resizer:v2.1.0",
"snapshotter": "registry.k8s.io/sig-storage/csi-snapshotter:v8.5.0",
"registrar": "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.17.0",
"snapshot-metadata": "registry.k8s.io/sig-storage/csi-snapshot-metadata:v1.0.0",
"plugin": "quay.io/cephcsi/cephcsi:v3.17.0",
"addons": "quay.io/csiaddons/k8s-sidecar:v0.14.0",
}
```
Meanwhile, `rook-ceph`'s chart (`deploy/charts/rook-ceph/templates/configmap.yaml`) unconditionally renders a ConfigMap named `rook-csi-operator-image-set-configmap`, populated from `.Values.csi.*` (the `rook-ceph` chart's own values), which currently defaults `csi.cephcsi.tag` to `v3.17.1`. On our cluster this ConfigMap is created correctly and contains the right values for every image *except* it's simply never read, because nothing points `OperatorConfig.spec.driverSpecDefaults.imageSet.name` (or any `Driver.spec.imageSet.name`) at it. The `ceph-csi-drivers` chart (v1.0.4, published at `https://ceph.github.io/ceph-csi-operator`) does not default `operatorConfig.driverSpecDefaults.imageSet.name` to that ConfigMap's name, so the `imageSetSpec.Name != ""` check is always false and the operator silently falls back to its own hardcoded `imageDefaults["plugin"]`.
Confirmed on the live cluster:
```
$ kubectl -n isak-ceph get cm rook-csi-operator-image-set-configmap -o yaml
data:
plugin: quay.io/cephcsi/cephcsi:v3.17.1
... (rest match imageDefaults exactly)
$ kubectl -n isak-ceph get operatorconfig ceph-csi-operator-config -o yaml
spec:
driverSpecDefaults:
# no imageSet field present at all
$ kubectl -n isak-ceph get driver isak-ceph.cephfs.csi.ceph.com -o jsonpath='{.spec.imageSet}'
# empty
$ kubectl -n isak-ceph get ds isak-ceph.cephfs.csi.ceph.com-nodeplugin -o jsonpath='{.spec.template.spec.containers[?(@.name=="csi-cephfsplugin")].image}'
quay.io/cephcsi/cephcsi:v3.17.0
```
Every other image in the ConfigMap (`provisioner`, `attacher`, `resizer`, `snapshotter`, `registrar`, `addons`) happens to match the operator's own compiled-in defaults exactly, which is what made this easy to miss — only `plugin`/`cephcsi` currently differs between the two, because `rook-ceph`'s chart default has moved ahead of `ceph-csi-operator`'s compiled default.
### Impact
Any install that relies on Helm chart defaults for `cephcsi`'s version (rather than explicitly setting `Driver.spec.imageSet` / `OperatorConfig.spec.driverSpecDefaults.imageSet`) silently runs a different `cephcsi` version than what the `rook-ceph` chart's documented default says it should. In a connected cluster this just means an unexpected extra pull (usually harmless); in an airgapped/offline install where only the chart-default-resolved images are pre-staged, it's a hard, permanent `ImagePullBackOff` with no obvious cause — the `ImageSet` ConfigMap that looks like it should be authoritative is entirely inert unless something explicitly wires it up.
### Expected behavior
Either:
1. The `ceph-csi-drivers` chart should default `operatorConfig.driverSpecDefaults.imageSet.name` to `rook-csi-operator-image-set-configmap` (matching what `rook-ceph`'s chart already creates for exactly this purpose — this is in fact what `rook/rook`'s own internal copy of a `ceph-csi-drivers`-shaped chart under `deploy/charts/ceph-csi-drivers` already defaults, for what it's worth, so the two charts appear to have diverged), or
2. `imageDefaults["plugin"]` in `internal/controller/defaults.go` should be kept in sync with whatever `rook-ceph`'s chart currently defaults `csi.cephcsi.tag` to, or
3. At minimum, the docs for `rook-ceph`'s CSI-Operator mode should call out that `csi.cephcsi.tag` on the `rook-ceph` release has no effect unless `imageSet` is explicitly wired up on the `ceph-csi-drivers` side.
### Environment
- Rook: v1.20.7 (`rook-ceph`, `rook-ceph-cluster` charts)
- ceph-csi-operator: v1.0.4 (`ceph-csi-drivers` chart, `https://ceph.github.io/ceph-csi-operator`)
- Install mode: CSI-Operator (`ROOK_V1_20_PLUS` path), airgapped/offline media build
- `helm list -n isak-ceph`:
```
ceph-csi-drivers 1 deployed ceph-csi-drivers-1.0.4 v1.0.4
rook-ceph 1 deployed rook-ceph-v1.20.7 v1.20.7
rook-ceph-cluster 1 deployed rook-ceph-cluster-v1.20.7 v1.20.7
```
### Workaround
Explicitly patch the `OperatorConfig` CR to point at the ConfigMap `rook-ceph`'s chart already creates:
```bash
kubectl -n patch operatorconfig ceph-csi-operator-config --type merge \
-p '{"spec":{"driverSpecDefaults":{"imageSet":{"name":"rook-csi-operator-image-set-configmap"}}}}'
kubectl -n rollout restart deploy/ceph-csi-controller-manager
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with internal/controller/driver_controller.go and internal/controller/defaults.go to trace image selection, then inspect deploy/charts/rook-ceph/templates/configmap.yaml and the ceph-csi-drivers chart defaults. Reproduce the default Helm installation and compare the rendered ConfigMap, OperatorConfig, and node-plugin images. Done means the intended default image source is wired consistently and a fresh default install uses the chart's v3.17.1 plugin image without a manual patch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, helm, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100