SolrBackup may never finish if REQUESTSTATUS returns notfound for an accepted backup
- Dominant language
- Go
- Stars
- 283
- Forks
- 148
- PR merge metrics
- No merged PRs in 30d
Description
# What happened
A `SolrBackup` can remain stuck in `InProgress` forever if:
1. the backup request is accepted ([operator backup submit](https://github.com/apache/solr-operator/blob/ca9d3c5c37a59f29570a6b49a8da5dc614aba75e/controllers/util/backup_util.go#L94-L109), [stable backup async ID](https://github.com/apache/solr-operator/blob/ca9d3c5c37a59f29570a6b49a8da5dc614aba75e/controllers/util/backup_util.go#L56-L58)),
2. the operator suddenly become unavailable,
3. the Solr async tracking entry for that backup is deleted ([DELETESTATUS API](https://github.com/apache/solr/blob/e0fe9619839f7ef2b43496104fa539a5b091db9a/solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java#L855-L930), [tracker deleteSingleAsyncId](https://github.com/apache/solr/blob/e0fe9619839f7ef2b43496104fa539a5b091db9a/solr/core/src/java/org/apache/solr/cloud/DistributedApiAsyncTracker.java#L284-L287)),
4. and the operator become available again.
After that:
- the backup files already exist in Solr,
- `REQUESTSTATUS` for the backup request ID returns `notfound` ([REQUESTSTATUS API](https://github.com/apache/solr/blob/e0fe9619839f7ef2b43496104fa539a5b091db9a/solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java#L793-L853), [tracker getAsyncTaskRequestStatus](https://github.com/apache/solr/blob/e0fe9619839f7ef2b43496104fa539a5b091db9a/solr/core/src/java/org/apache/solr/cloud/DistributedApiAsyncTracker.java#L217-L270)),
- but the `SolrBackup` CR still stays in `inProgress=true`,
- and the operator keeps polling instead of reaching a terminal state ([backup reconcile loop](https://github.com/apache/solr-operator/blob/ca9d3c5c37a59f29570a6b49a8da5dc614aba75e/controllers/solrbackup_controller.go#L284-L306), [requeue after 5s](https://github.com/apache/solr-operator/blob/ca9d3c5c37a59f29570a6b49a8da5dc614aba75e/controllers/solrbackup_controller.go#L141-L143)).
# Environment
- macOS
- local `kind` cluster
- Kubernetes / kind node version: `v1.32.1`
- `solr-operator` version: `v0.10.0-orerekease`
- `solr-operator` built from `master` on March 22, 2026 (`ca9d3c5c37a59f29570a6b49a8da5dc614aba75e`)
- Solr version: `9.10.0`
# Steps to reproduce
1. Deploy `solr-operator` on a local `kind` cluster.
2. Create a 1-node `SolrCloud` with a local backup repository, then create a
collection and start a `SolrBackup` for it.
3. As soon as the backup first shows `inProgress=true`, scale the
`solr-operator` deployment down to `0`.
4. While the operator is down, wait for the Solr async request to finish, then
delete only that async status entry with `DELETESTATUS`.
5. Confirm the backup data still exists, but `REQUESTSTATUS` for that same
request ID now returns `notfound`.
6. Scale the operator back up to `1` and observe that the `SolrBackup` CR never
reaches a terminal state.
The stuck status looks like:
```yaml
status:
collectionBackupStatuses:
- asyncBackupStatus: notfound
inProgress: true
```
and it never sets `finished: true` or `successful: true`.
# Expected behavior
Once an accepted backup later becomes `notfound`, the operator should not leave the CR in `InProgress` forever.
It should eventually either:
- recover, or
- mark the backup failed with a clear reason.
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the accepted-backup scenario, then read controllers/util/backup_util.go and controllers/solrbackup_controller.go around request submission, status handling, and the five-second requeue. Trace how REQUESTSTATUS=notfound is processed; done means the SolrBackup reaches a terminal state with a clear recovery or failure outcome instead of remaining in progress forever.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- infrastructure, search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100