Cannot upgrade from `1.9/stable` to `1.9/candidate`

### Bug Description

Issue coming from labels change in #295.
Upgrade from `rev704` (current `1.9/stable`) to `rev732` (current `1.9/candidate`) fails.
Current stable revision adds the following labels:
```yaml
labels:
app.kubernetes.io/name: training-operator-manager
control-plane: kubeflow-training-operator
```
while current candidate only has `control-plane` one. In an already existing deployment, editing labels is not allowed[^1].

[^1]: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#label-selector-updates

### To Reproduce

1. Deploy `training-operator` from `1.9/stable`
2. Upgrade to `1.9/candidate`

### Environment

.

### Relevant Log Output

```shell
2026-09-01T07:43:56.807Z [container-agent] 2026-09-01 07:43:56 INFO juju-log HTTP Request: PATCH https://10.152.183.1/apis/apps/v1/namespaces/kubeflow/deployments/training-operator?force=true&fieldManager=lightkube "HTTP/1.1 422 Unprocessable Entity"
2026-09-01T07:43:56.825Z [container-agent] 2026-09-01 07:43:56 ERROR juju-log Uncaught exception while in charm code:
2026-09-01T07:43:56.825Z [container-agent] Traceback (most recent call last):
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/lightkube/core/generic_client.py", line 235, in raise_for_status
2026-09-01T07:43:56.825Z [container-agent] resp.raise_for_status()
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/httpx/_models.py", line 763, in raise_for_status
2026-09-01T07:43:56.825Z [container-agent] raise HTTPStatusError(message, request=request, response=self)
2026-09-01T07:43:56.825Z [container-agent] httpx.HTTPStatusError: Client error '422 Unprocessable Entity' for url 'https://10.152.183.1/apis/apps/v1/namespaces/kubeflow/deployments/training-operator?force=true&fieldManager=lightkube'
2026-09-01T07:43:56.825Z [container-agent] For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/422
2026-09-01T07:43:56.825Z [container-agent]
2026-09-01T07:43:56.825Z [container-agent] During handling of the above exception, another exception occurred:
2026-09-01T07:43:56.825Z [container-agent]
2026-09-01T07:43:56.825Z [container-agent] Traceback (most recent call last):
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/src/charm.py", line 252, in _apply_k8s_resources
2026-09-01T07:43:56.825Z [container-agent] self.k8s_resource_handler.apply()
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/kubernetes/_kubernetes_resource_handler.py", line 363, in apply
2026-09-01T07:43:56.825Z [container-agent] raise e
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/kubernetes/_kubernetes_resource_handler.py", line 336, in apply
2026-09-01T07:43:56.825Z [container-agent] apply_many(
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/lightkube/batch/_many.py", line 76, in apply_many
2026-09-01T07:43:56.825Z [container-agent] returns[i] = client.apply(
2026-09-01T07:43:56.825Z [container-agent] ^^^^^^^^^^^^^
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/lightkube/core/client.py", line 757, in apply
2026-09-01T07:43:56.825Z [container-agent] return self.patch(
2026-09-01T07:43:56.825Z [container-agent] ^^^^^^^^^^^
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/lightkube/core/client.py", line 492, in patch
2026-09-01T07:43:56.825Z [container-agent] return self._client.request(
2026-09-01T07:43:56.825Z [container-agent] ^^^^^^^^^^^^^^^^^^^^^
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/lightkube/core/generic_client.py", line 311, in request
2026-09-01T07:43:56.825Z [container-agent] return self.handle_response(method, resp, br)
2026-09-01T07:43:56.825Z [container-agent] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/lightkube/core/generic_client.py", line 251, in handle_response
2026-09-01T07:43:56.825Z [container-agent] self.raise_for_status(resp)
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/lightkube/core/generic_client.py", line 237, in raise_for_status
2026-09-01T07:43:56.825Z [container-agent] raise transform_exception(e)
2026-09-01T07:43:56.825Z [container-agent] lightkube.core.exceptions.ApiError: Deployment.apps "training-operator" is invalid: spec.selector: Invalid value: {"matchLabels":{"control-plane":"kubeflow-training-operator"}}: field is immutable
2026-09-01T07:43:56.825Z [container-agent]
2026-09-01T07:43:56.825Z [container-agent] The above exception was the direct cause of the following exception:
2026-09-01T07:43:56.825Z [container-agent]
2026-09-01T07:43:56.825Z [container-agent] Traceback (most recent call last):
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/src/charm.py", line 320, in
2026-09-01T07:43:56.825Z [container-agent] main(TrainingOperatorCharm)
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/ops/__init__.py", line 356, in __call__
2026-09-01T07:43:56.825Z [container-agent] return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
2026-09-01T07:43:56.825Z [container-agent] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 502, in main
2026-09-01T07:43:56.825Z [container-agent] manager.run()
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 486, in run
2026-09-01T07:43:56.825Z [container-agent] self._emit()
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 421, in _emit
2026-09-01T07:43:56.825Z [container-agent] self._emit_charm_event(self.dispatcher.event_name)
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 465, in _emit_charm_event
2026-09-01T07:43:56.825Z [container-agent] event_to_emit.emit(*args, **kwargs)
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 351, in emit
2026-09-01T07:43:56.825Z [container-agent] framework._emit(event)
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 924, in _emit
2026-09-01T07:43:56.825Z [container-agent] self._reemit(event_path)
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 1030, in _reemit
2026-09-01T07:43:56.825Z [container-agent] custom_handler(event)
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/src/charm.py", line 300, in _on_upgrade
2026-09-01T07:43:56.825Z [container-agent] self._on_event(_, force_conflicts=True)
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/src/charm.py", line 278, in _on_event
2026-09-01T07:43:56.825Z [container-agent] self._apply_k8s_resources(force_conflicts=force_conflicts)
2026-09-01T07:43:56.825Z [container-agent] File "/var/lib/juju/agents/unit-training-operator-0/charm/src/charm.py", line 261, in _apply_k8s_resources
2026-09-01T07:43:56.825Z [container-agent] raise GenericCharmRuntimeError("K8S resources creation failed") from error
2026-09-01T07:43:56.825Z [container-agent] .GenericCharmRuntimeError: K8S resources creation failed
2026-09-01T07:43:57.075Z [container-agent] 2026-09-01 07:43:57 ERROR juju.worker.uniter.operation runhook.go:180 hook "upgrade-charm" (via hook dispatching script: dispatch) failed: exit status 1
```

### Additional Context

_No response_

Start by reading charm/src/charm.py, especially _on_upgrade and _apply_k8s_resources, then reproduce the upgrade from rev704 to rev732. Compare the Deployment labels and selector involved in the two revisions, using the reported 422 error as the guide. Done means an existing 1.9/stable deployment upgrades to 1.9/candidate without the immutable selector failure.

Written by the indexing model from the issue text.