Scale down operation fails and is never requeued if `getReplicasForPod` fails transiently
- Dominant language
- Go
- Stars
- 283
- Forks
- 148
- PR merge metrics
- No merged PRs in 30d
Description
## Bug Report
**What did you do? / Minimal Reproducible Example**
We first deployed a five-replica Solr cluster, then modified the CR to scale down to three replicas.
The scale down triggers the solr-operator to perform the ManagedScaleDown operation, invoking the `handleManagedCloudScaleDown` function. Inside the function, the operator invokes the `evictSinglePod` function. If the call to the Solr transiently fails (due to transient network issue or transient unavailability of Solr), the operator writes an error message and never retries. Even if the transient fault is resolved later, the operator is stuck and never perform the scale down operation.
To reproduce this bug,
1. apply the CR to deploy a 5 replica Solr cluster by changing replicas to 5 in the example: https://github.com/apache/solr-operator/blob/main/example/test_solrcloud.yaml
2. Expose faults to the operator (this can be reliably reproduced using the ChaosMesh controller by injecting a network loss between the Solr pods and the Solr operator)
3. Change the `replicas: 5` to `replicas:3` in the CR to initiate the scale down.
4. Observe that the operator reports an error log because it is unable to contact the Solr Pods for status
```
ERROR Error retrieving cluster status, cannot determine if pod has replicas {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"example","namespace":"default"}, "namespace": "default", "name": "example", "reconcileID": "66d08e1c-a6cd-4176-a792-c706177cc39f", "error": "Get \"http://example-solrcloud-common.default/solr/admin/collections?action=CLUSTERSTATUS&wt=json\": dial tcp 10.96.177.119:80: i/o timeout"}
```
5. Remove the fault and observe that the operator never requeues and reconciles the request
**What did you expect to see?**
The operator should be able to scale down the cluster after the transient faults go away
**What did you see instead?**
The operator never requeues the request
**Root Cause**
The root cause of this bug is at https://github.com/apache/solr-operator/blob/b2c951052828f88e9fc415f54aec189b805117e9/controllers/solr_cluster_ops_util.go#L251C57-L251C71
where the operator does not handle the error returned by `evictSinglePod` but just directly returns normally.
The fix should be simple to add an error handling branch to requeue the request.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in controllers/solr_cluster_ops_util.go at the reported line in the ManagedScaleDown path, following handleManagedCloudScaleDown into evictSinglePod. Verify how the transient error is currently returned and how reconciliation requeue behavior is handled elsewhere. Done means a failed pod eviction causes the request to requeue and scaling from five to three replicas completes after the Solr fault is removed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- infrastructure, search
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 50/100