A failed status update leaves the SolrBackup permanently stuck in progress
- Dominant language
- Go
- Stars
- 283
- Forks
- 148
- PR merge metrics
- No merged PRs in 30d
Description
# Environment
- solr-operator built from `ed5c5c7d28a4c1189d19f581259e05385c0d4b20`
- Solr 9.7.0
- Kubernetes v1.35.0 (kind v0.31.0, 3 nodes)
- A `SolrBackup` against a healthy single-node SolrCloud
# What happened
A `SolrBackup` submitted a backup to Solr, but the operator failed to save `InProgress=true` to the apiserver. The backup then remained stuck.
This happens if the operator restarts after Solr accepts the backup but before the status update is saved. After the restart, the operator reads `InProgress=false` and submits the same backup again. Solr rejects the duplicate async ID, so the operator never polls or cleans up the original request.
The backup only recovered after I manually called `DELETESTATUS`. The next submission then succeeded and the backup completed.
# Where the source code is wrong
[`reconcileSolrCollectionBackup`](https://github.com/apache/solr-operator/blob/ed5c5c7d28a4c1189d19f581259e05385c0d4b20/controllers/solrbackup_controller.go#L285-L300) decides what to do from the saved `InProgress` value. If the value is `false`, it submits a backup without first checking Solr:
```go
// controllers/solrbackup_controller.go:285-300
if collectionBackupStatus.Finished {
return true, nil
} else if !collectionBackupStatus.InProgress {
started, err = util.StartBackupForCollection(...) // no pre-check
if err != nil {
return true, err
}
collectionBackupStatus.InProgress = started // in-memory only
...
} else if collectionBackupStatus.InProgress {
// REQUESTSTATUS poll, and DELETESTATUS cleanup on finish
}
```
`InProgress` is only persisted [at the end of `Reconcile`](https://github.com/apache/solr-operator/blob/ed5c5c7d28a4c1189d19f581259e05385c0d4b20/controllers/solrbackup_controller.go#L178-L181):
```go
// controllers/solrbackup_controller.go:178-181
if !reflect.DeepEqual(unmodifiedBackupResource.Status, backup.Status) {
err = r.Status().Patch(ctx, backup, client.MergeFrom(unmodifiedBackupResource))
}
```
If this patch does not complete after Solr accepts the backup, etcd still contains `InProgress=false`. Every later reconcile submits the same async ID. Solr rejects it, and the function returns before setting `InProgress=true`. This repeats indefinitely.
Solr keeps completed async records until `DELETESTATUS` is called. However, the operator only calls `DELETESTATUS` when `InProgress=true`, so it cannot clean up the record.
The [cluster-operation code](https://github.com/apache/solr-operator/blob/ed5c5c7d28a4c1189d19f581259e05385c0d4b20/controllers/util/solr_update_util.go#L564-L570) avoids this problem by checking Solr before submitting:
```go
// controllers/util/solr_update_util.go:564-570
// First check to see if the Async Replace request has started
if asyncState, message, asyncErr := solr_api.CheckAsyncRequest(ctx, solrCloud, requestId); asyncErr != nil {
...
} else if asyncState == "notfound" {
// Submit new Replace Node request
```
This code can detect an existing request even if an operator status update was lost. The backup code does not perform this check.
# Expected behavior
A failed status update should not leave the backup stuck after Solr has accepted it.
Before submitting, the backup code should call `CheckAsyncRequest` with the async ID:
- `notfound` → submit the backup
- running / completed / failed → set `InProgress=true` and use the existing polling path
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in controllers/solrbackup_controller.go at reconcileSolrCollectionBackup and inspect the existing CheckAsyncRequest usage in controllers/util/solr_update_util.go. Before submission, account for an existing async request and route it through the polling path. Done means a lost status update no longer causes duplicate submission, and running, completed, or failed requests can be recovered and cleaned up.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend-api-design, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100