kubeflow / kubeflow/notebooks

bug: StatefulSet spec.selector is immutable and cannot be updated in CopyStatefulSetFields

Open
#1,377 5 comments 0 reactions 1 assignee Claimed by @prdai View on GitHub
Dominant language
No language data
Stars
84
Forks
149
Avg merge
5d 15h
Merged PRs (30d)
29

Description

### Description

In `workspaces/controller/internal/helper/helper.go`, the `CopyStatefulSetFields` function attempts to copy and update `spec.selector` on existing StatefulSets:

```go
// copy `spec.selector`
//
// TODO: confirm if StatefulSets support updates to the selector
// if not, we might need to recreate the StatefulSet
//
if !equality.Semantic.DeepEqual(target.Spec.Selector, desired.Spec.Selector) {
target.Spec.Selector = desired.Spec.Selector
requireUpdate = true
}
```

However, Kubernetes StatefulSets do not support updates to `spec.selector` because `spec.selector` is immutable after creation.

### Kubernetes Reference

According to upstream Kubernetes [`staging/src/k8s.io/api/apps/v1/types.go#L214`](https://github.com/kubernetes/kubernetes/blob/73b08be6346d68bf550c95789fdde1a305dd0820/staging/src/k8s.io/api/apps/v1/types.go#L214) and API validation in `pkg/apis/apps/validation/validation.go`:
- StatefulSet `spec.selector` is an immutable field (`apivalidation.ValidateImmutableField(...)`).
- Any attempt to modify `spec.selector` in an update request is rejected by the Kubernetes API server with a 422 Invalid error: `spec.selector: Invalid value: ...: field is immutable`.

### Impact

When `CopyStatefulSetFields` detects a difference in `target.Spec.Selector` vs `desired.Spec.Selector`, it sets `target.Spec.Selector = desired.Spec.Selector` and returns `requireUpdate = true`.

When `workspace_controller.go` attempts to apply this update via `r.Update(ctx, foundStatefulSet)`:
```go
if helper.CopyStatefulSetFields(statefulSet, foundStatefulSet) {
if err := r.Update(ctx, foundStatefulSet); err != nil {
log.Error(err, "unable to update StatefulSet")
return ctrl.Result{}, err
}
}
```
The API server rejects the update, causing the controller to enter a failing reconciliation loop.

### Suggested Fix

1. Remove updating `target.Spec.Selector` in `CopyStatefulSetFields`, as in-place updates to `spec.selector` are not allowed by the Kubernetes API server.
2. If selector updates need to be supported, the controller should recreate (delete and create) the StatefulSet.
3. If selector updates are not supported, consider detecting selector mismatches and logging an appropriate error or handling recreation rather than attempting an invalid `r.Update`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.