apache / apache/solr-operator

A failed status update leaves the SolrBackup permanently stuck in progress

Open
#848 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
283
Forks
148
PR merge metrics
No merged PRs in 30d

Description

# Environment

- solr-operator built from `ed5c5c7d28a4c1189d19f581259e05385c0d4b20`
- Solr 9.7.0
- Kubernetes v1.35.0 (kind v0.31.0, 3 nodes)
- A `SolrBackup` against a healthy single-node SolrCloud

# What happened

A `SolrBackup` submitted a backup to Solr, but the operator failed to save `InProgress=true` to the apiserver. The backup then remained stuck.

This happens if the operator restarts after Solr accepts the backup but before the status update is saved. After the restart, the operator reads `InProgress=false` and submits the same backup again. Solr rejects the duplicate async ID, so the operator never polls or cleans up the original request.

The backup only recovered after I manually called `DELETESTATUS`. The next submission then succeeded and the backup completed.

# Where the source code is wrong

[`reconcileSolrCollectionBackup`](https://github.com/apache/solr-operator/blob/ed5c5c7d28a4c1189d19f581259e05385c0d4b20/controllers/solrbackup_controller.go#L285-L300) decides what to do from the saved `InProgress` value. If the value is `false`, it submits a backup without first checking Solr:

```go
// controllers/solrbackup_controller.go:285-300
if collectionBackupStatus.Finished {
return true, nil
} else if !collectionBackupStatus.InProgress {
started, err = util.StartBackupForCollection(...) // no pre-check
if err != nil {
return true, err
}
collectionBackupStatus.InProgress = started // in-memory only
...
} else if collectionBackupStatus.InProgress {
// REQUESTSTATUS poll, and DELETESTATUS cleanup on finish
}
```

`InProgress` is only persisted [at the end of `Reconcile`](https://github.com/apache/solr-operator/blob/ed5c5c7d28a4c1189d19f581259e05385c0d4b20/controllers/solrbackup_controller.go#L178-L181):

```go
// controllers/solrbackup_controller.go:178-181
if !reflect.DeepEqual(unmodifiedBackupResource.Status, backup.Status) {
err = r.Status().Patch(ctx, backup, client.MergeFrom(unmodifiedBackupResource))
}
```

If this patch does not complete after Solr accepts the backup, etcd still contains `InProgress=false`. Every later reconcile submits the same async ID. Solr rejects it, and the function returns before setting `InProgress=true`. This repeats indefinitely.

Solr keeps completed async records until `DELETESTATUS` is called. However, the operator only calls `DELETESTATUS` when `InProgress=true`, so it cannot clean up the record.

The [cluster-operation code](https://github.com/apache/solr-operator/blob/ed5c5c7d28a4c1189d19f581259e05385c0d4b20/controllers/util/solr_update_util.go#L564-L570) avoids this problem by checking Solr before submitting:

```go
// controllers/util/solr_update_util.go:564-570
// First check to see if the Async Replace request has started
if asyncState, message, asyncErr := solr_api.CheckAsyncRequest(ctx, solrCloud, requestId); asyncErr != nil {
...
} else if asyncState == "notfound" {
// Submit new Replace Node request
```

This code can detect an existing request even if an operator status update was lost. The backup code does not perform this check.

# Expected behavior

A failed status update should not leave the backup stuck after Solr has accepted it.

Before submitting, the backup code should call `CheckAsyncRequest` with the async ID:

- `notfound` → submit the backup
- running / completed / failed → set `InProgress=true` and use the existing polling path

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in controllers/solrbackup_controller.go at reconcileSolrCollectionBackup and inspect the existing CheckAsyncRequest usage in controllers/util/solr_update_util.go. Before submission, account for an existing async request and route it through the polling path. Done means a lost status update no longer causes duplicate submission, and running, completed, or failed requests can be recovered and cleaned up.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend-api-design, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.