SolrCloud Pod moved to new Node - Replica Migration pending

Environment:
Solr Operator Helm : 0.8.0
Solr 9.4 container image
3 node cluster
Persistent storage option (w/ localvolume provisioner)
Managed upgrade strategy.

The K8S node for solrcloud-0 got cordon and Pod was moved to a new node.

When the pod came up on new node, its recognized by as part of the SolrCloud statefulset, but at the Collection level replica was lost on the node. Looking at the Stateful , there's a cluster lock.

solr.apache.org/clusterOpsLock: >-
{"operation":"RollingUpdate","lastStartTime":"2023-12-14T22:08:47Z","metadata":"{\"requiresReplicaMigration\":false}"}

Plz allow me to ask if im missing a step in the process...

**Should the operator automatically do the replica migration?**
I have read about Rebalance API and using 9.4 version.

**Is there a way to manually kick off the replica migration step to that specific POD?**

SolrCloud custom definition.

```
apiVersion: solr.apache.org/v1beta1
kind: SolrCloud
metadata:
annotations:
meta.helm.sh/release-name: solr
meta.helm.sh/release-namespace: solr
creationTimestamp: '2023-12-14T21:10:55Z'
finalizers:
- storage.finalizers.solr.apache.org
generation: 3
labels:
app.kubernetes.io/instance: solr
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: solr
app.kubernetes.io/version: 8.11.1
helm.sh/chart: solr-0.8.0
managedFields:
- apiVersion: solr.apache.org/v1beta1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
...
--removed metadata here for readability
...
manager: solr-operator
operation: Update
time: '2023-12-14T21:10:55Z'
- apiVersion: solr.apache.org/v1beta1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
...
--removed metadata here for readability
...
manager: helm
operation: Update
time: '2023-12-14T21:58:46Z'
- apiVersion: solr.apache.org/v1beta1
fieldsType: FieldsV1
fieldsV1:
f:status:
...
---removed fields{} for readability
...
manager: solr-operator
operation: Update
subresource: status
time: '2023-12-14T21:59:05Z'
name: solr
namespace: solr
resourceVersion: '115467'
uid: 559977d2-2fd0-42fa-bf28-08bc5cebf851
selfLink: /apis/solr.apache.org/v1beta1/namespaces/solr/solrclouds/solr
status:
externalCommonAddress: http://solr-solr-solrcloud.k8s.solr.cloud
internalCommonAddress: http://solr-solrcloud-common.solr
podSelector: solr-cloud=solr,technology=solr-cloud
readyReplicas: 3
replicas: 3
solrNodes:
- externalAddress: http://solr-solr-solrcloud-0.k8s.solr.cloud
internalAddress: http://solr-solrcloud-0.solr
name: solr-solrcloud-0
nodeName: ip-x-y-162-17.us-west-2.compute.internal
ready: true
scheduledForDeletion: false
specUpToDate: true
version: '0.8'
- externalAddress: http://solr-solr-solrcloud-1.k8s.solr.cloud
internalAddress: http://solr-solrcloud-1.solr
name: solr-solrcloud-1
nodeName: ip-x-y-160-139.us-west-2.compute.internal
ready: true
scheduledForDeletion: false
specUpToDate: false
version: '0.8'
- externalAddress: http://solr-solr-solrcloud-2.k8s.solr.cloud
internalAddress: http://solr-solrcloud-2.solr
name: solr-solrcloud-2
nodeName: ip-x-y-163-213.us-west-2.compute.internal
ready: true
scheduledForDeletion: false
specUpToDate: false
version: '0.8'
upToDateNodes: 1
version: '0.8'
zookeeperConnectionInfo:
chroot: /
externalConnectionString: N/A
internalConnectionString: >-
solr-solrcloud-zookeeper-0.solr-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181,solr-solrcloud-zookeeper-1.solr-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181,solr-solrcloud-zookeeper-2.solr-solrcloud-zookeeper-headless.solr.svc.cluster.local:2181
spec:
availability:
podDisruptionBudget:
enabled: true
method: ClusterWide
busyBoxImage:
repository: library/busybox
tag: 1.28.0-glibc
customSolrKubeOptions:
podOptions:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: technology
operator: In
values:
- solr-cloud
- key: solr-cloud
operator: In
values:
- solr
topologyKey: topology.kubernetes.io/zone
weight: 100
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: technology
operator: In
values:
- solr-cloud
- key: solr-cloud
operator: In
values:
- solr
topologyKey: kubernetes.io/hostname
annotations:
manualrestart: '2023-12-14T00:00:01Z'
defaultInitContainerResources: {}
resources:
limits:
cpu: '16'
memory: 32G
requests:
cpu: '8'
memory: 16G
serviceAccountName: solr-operator
tolerations:
- effect: NoSchedule
key: role
operator: Equal
value: solr-cluster
dataStorage:
persistent:
pvcTemplate:
metadata: {}
spec:
resources:
requests:
storage: 500Gi
storageClassName: my-disks
reclaimPolicy: Delete
replicas: 3
scaling:
populatePodsOnScaleUp: true
vacatePodsOnScaleDown: true
solrAddressability:
commonServicePort: 80
external:
domainName: k8s.solr.cloud
method: Ingress
nodePortOverride: 80
useExternalAddress: false
podPort: 8983
solrImage:
pullPolicy: Always
repository: mycustom.registry.builton.9-4solr
tag: latest
solrJavaMem: '-Xms8192m -Xmx16384m'
solrLogLevel: INFO
solrOpts: '-Denable.runtime.lib=true -Denable.packages=true'
updateStrategy:
managed: {}
method: Managed
zookeeperRef:
provided:
adminServerService: {}
chroot: /
clientService: {}
config: {}
ephemeral:
emptydirvolumesource: {}
headlessService: {}
image:
pullPolicy: IfNotPresent
repository: pravega/zookeeper
maxUnavailableReplicas: 1
replicas: 3
zookeeperPodPolicy:
resources: {}
serviceAccountName: solr-operator

```

Start by reviewing the SolrCloud custom definition, the clusterOpsLock metadata, and the managed RollingUpdate state described in the report. Reproduce the node-cordon and pod-move scenario with Solr 9.4 and check the replica state and available Rebalance API behavior. Done means establishing whether migration is expected automatically or can be triggered for the affected pod, then documenting or fixing the confirmed behavior.

Written by the indexing model from the issue text.