ceph / ceph/ceph-csi-operator

## `ceph-csi-operator` falls back to its hardcoded `cephcsi:v3.17.0` default instead of the rook-ceph chart's `v3.17.1`, because the `ceph-csi-drivers` chart never sets `operatorConfig.driverSpecDefaults.imageSet.name`

Closed
#605 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
63
Forks
59
Avg merge
1d 19h
Merged PRs (30d)
19

Description

## `ceph-csi-operator` falls back to its hardcoded `cephcsi:v3.17.0` default instead of the rook-ceph chart's `v3.17.1`, because the `ceph-csi-drivers` chart never sets `operatorConfig.driverSpecDefaults.imageSet.name`

### Summary

On a fresh Rook v1.20.7 install (using the CSI-Operator path: `rook-ceph` chart + `ceph-csi-drivers` chart v1.0.4, both installed via `helm upgrade --install` with no custom `csi.*` overrides beyond the chart defaults), the `csi-cephfsplugin` / `csi-rbdplugin` node-plugin containers end up pinned to `quay.io/cephcsi/cephcsi:v3.17.0`, while the `rook-ceph` chart's own default (`csi.cephcsi.tag`) is `v3.17.1`. In an airgapped/offline environment where only the images referenced by the Helm chart defaults were pre-pulled (i.e. `v3.17.1`), this produces a permanent `ImagePullBackOff` on every node.

### Root cause

`internal/controller/driver_controller.go` in `ceph-csi-operator` builds its working image map like this:

```go
r.images = maps.Clone(imageDefaults) // internal/controller/defaults.go

imageSetSpec := opConfig.Spec.DriverSpecDefaults.ImageSet
if imageSetSpec != nil && imageSetSpec.Name != "" {
// load ConfigMap and merge over defaults
maps.Copy(r.images, imageSetCM.Data)
}

imageSetSpec := r.driver.Spec.ImageSet
if imageSetSpec != nil && imageSetSpec.Name != "" {
maps.Copy(r.images, imageSetCM.Data)
}
```

`internal/controller/defaults.go` (still true on `main` as of writing) hardcodes:

```go
var imageDefaults = map[string]string{
"provisioner": "registry.k8s.io/sig-storage/csi-provisioner:v6.2.0",
"attacher": "registry.k8s.io/sig-storage/csi-attacher:v4.12.0",
"resizer": "registry.k8s.io/sig-storage/csi-resizer:v2.1.0",
"snapshotter": "registry.k8s.io/sig-storage/csi-snapshotter:v8.5.0",
"registrar": "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.17.0",
"snapshot-metadata": "registry.k8s.io/sig-storage/csi-snapshot-metadata:v1.0.0",
"plugin": "quay.io/cephcsi/cephcsi:v3.17.0",
"addons": "quay.io/csiaddons/k8s-sidecar:v0.14.0",
}
```

Meanwhile, `rook-ceph`'s chart (`deploy/charts/rook-ceph/templates/configmap.yaml`) unconditionally renders a ConfigMap named `rook-csi-operator-image-set-configmap`, populated from `.Values.csi.*` (the `rook-ceph` chart's own values), which currently defaults `csi.cephcsi.tag` to `v3.17.1`. On our cluster this ConfigMap is created correctly and contains the right values for every image *except* it's simply never read, because nothing points `OperatorConfig.spec.driverSpecDefaults.imageSet.name` (or any `Driver.spec.imageSet.name`) at it. The `ceph-csi-drivers` chart (v1.0.4, published at `https://ceph.github.io/ceph-csi-operator`) does not default `operatorConfig.driverSpecDefaults.imageSet.name` to that ConfigMap's name, so the `imageSetSpec.Name != ""` check is always false and the operator silently falls back to its own hardcoded `imageDefaults["plugin"]`.

Confirmed on the live cluster:

```
$ kubectl -n isak-ceph get cm rook-csi-operator-image-set-configmap -o yaml
data:
plugin: quay.io/cephcsi/cephcsi:v3.17.1
... (rest match imageDefaults exactly)

$ kubectl -n isak-ceph get operatorconfig ceph-csi-operator-config -o yaml
spec:
driverSpecDefaults:
# no imageSet field present at all

$ kubectl -n isak-ceph get driver isak-ceph.cephfs.csi.ceph.com -o jsonpath='{.spec.imageSet}'
# empty

$ kubectl -n isak-ceph get ds isak-ceph.cephfs.csi.ceph.com-nodeplugin -o jsonpath='{.spec.template.spec.containers[?(@.name=="csi-cephfsplugin")].image}'
quay.io/cephcsi/cephcsi:v3.17.0
```

Every other image in the ConfigMap (`provisioner`, `attacher`, `resizer`, `snapshotter`, `registrar`, `addons`) happens to match the operator's own compiled-in defaults exactly, which is what made this easy to miss — only `plugin`/`cephcsi` currently differs between the two, because `rook-ceph`'s chart default has moved ahead of `ceph-csi-operator`'s compiled default.

### Impact

Any install that relies on Helm chart defaults for `cephcsi`'s version (rather than explicitly setting `Driver.spec.imageSet` / `OperatorConfig.spec.driverSpecDefaults.imageSet`) silently runs a different `cephcsi` version than what the `rook-ceph` chart's documented default says it should. In a connected cluster this just means an unexpected extra pull (usually harmless); in an airgapped/offline install where only the chart-default-resolved images are pre-staged, it's a hard, permanent `ImagePullBackOff` with no obvious cause — the `ImageSet` ConfigMap that looks like it should be authoritative is entirely inert unless something explicitly wires it up.

### Expected behavior

Either:
1. The `ceph-csi-drivers` chart should default `operatorConfig.driverSpecDefaults.imageSet.name` to `rook-csi-operator-image-set-configmap` (matching what `rook-ceph`'s chart already creates for exactly this purpose — this is in fact what `rook/rook`'s own internal copy of a `ceph-csi-drivers`-shaped chart under `deploy/charts/ceph-csi-drivers` already defaults, for what it's worth, so the two charts appear to have diverged), or
2. `imageDefaults["plugin"]` in `internal/controller/defaults.go` should be kept in sync with whatever `rook-ceph`'s chart currently defaults `csi.cephcsi.tag` to, or
3. At minimum, the docs for `rook-ceph`'s CSI-Operator mode should call out that `csi.cephcsi.tag` on the `rook-ceph` release has no effect unless `imageSet` is explicitly wired up on the `ceph-csi-drivers` side.

### Environment

- Rook: v1.20.7 (`rook-ceph`, `rook-ceph-cluster` charts)
- ceph-csi-operator: v1.0.4 (`ceph-csi-drivers` chart, `https://ceph.github.io/ceph-csi-operator`)
- Install mode: CSI-Operator (`ROOK_V1_20_PLUS` path), airgapped/offline media build
- `helm list -n isak-ceph`:
```
ceph-csi-drivers 1 deployed ceph-csi-drivers-1.0.4 v1.0.4
rook-ceph 1 deployed rook-ceph-v1.20.7 v1.20.7
rook-ceph-cluster 1 deployed rook-ceph-cluster-v1.20.7 v1.20.7
```

### Workaround

Explicitly patch the `OperatorConfig` CR to point at the ConfigMap `rook-ceph`'s chart already creates:

```bash
kubectl -n patch operatorconfig ceph-csi-operator-config --type merge \
-p '{"spec":{"driverSpecDefaults":{"imageSet":{"name":"rook-csi-operator-image-set-configmap"}}}}'
kubectl -n rollout restart deploy/ceph-csi-controller-manager
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with internal/controller/driver_controller.go and internal/controller/defaults.go to trace image selection, then inspect deploy/charts/rook-ceph/templates/configmap.yaml and the ceph-csi-drivers chart defaults. Reproduce the default Helm installation and compare the rendered ConfigMap, OperatorConfig, and node-plugin images. Done means the intended default image source is wired consistently and a fresh default install uses the chart's v3.17.1 plugin image without a manual patch.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, helm, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.