restore_check reports success when failed shards disappear from _recovery (silent false success)
- Dominant language
- Python
- Stars
- 3.1k
- Forks
- 627
- PR merge metrics
- No merged PRs in 30d
Description
**Expected Behavior**:
When a snapshot restore fails (a shard cannot be recovered, e.g. due to `IOException: No space left on device`), Curator's restore action should report a failure — or at least not report success.
**Actual Behavior**:
`restore_check` decides the restore is complete by polling `client.indices.recovery()` and returning True once every shard present in the response reports `stage: DONE`:
- master: `curator/helpers/waiters.py`, `restore_check` (~lines 141-189)(https://github.com/elastic/curator/blob/master/curator/helpers/waiters.py#L141-L189)
- older 5.x: `curator/utils.py`, lines 1744-1756 (https://github.com/elastic/curator/blob/v5.8.4/curator/utils.py)
The **trap**: if all the shards in the response show DONE, that does not automatically mean the restore finished successfully. Why? Because shards that fail disappear from the response (the shard is deallocated — UNASSIGNED/ALLOCATION_FAILED — and drops out of the /_recovery response entirely). So if a shard failed, you will not get that information from this endpoint. It will tell you that all the remaining shards are DONE, but that does not mean the restore succeeded.
## Steps to Reproduce
1. Snapshot indices larger than the free disk space on the target node
(in our case ~82 GB snapshot vs ~80 GB free).
2. Restore via Curator's restore action with `wait_for_completion: True`
(Curator-level waiting, i.e. the `restore_check` polling loop).
3. One shard hits `IOException: No space left on device`; Elasticsearch
retries allocation, gives up, and the shard leaves `_recovery`.
4. `restore_check` will see only the remaining shards, all with `stage: DONE`,
and return True — Curator reports success while `GET /_cluster/health` is red.
Note: we hit this bug in our own application, which uses the same
`_recovery`-based check; the Curator behavior is established by code
inspection of `restore_check`, not by a Curator run.
## Suggested Fix
After the all-DONE condition, verify the outcome before declaring success:
`GET /_cluster/health` for the restored indices (green/yellow → success),
and on `red` inspect `GET /_cluster/allocation/explain` for the unassigned
primary (`can_allocate: no` → fail with the reason from
`unassigned_info.details`).
## Context
- Same trap reported against the Elasticsearch docs:
elastic/elasticsearch#157608
- Kibana's Snapshot & Restore UI has the same issue: elastic/kibana#63154
- Versions: Curator behavior verified in master and v5.8.4;
Elasticsearch 7.17.27; RHEL 9.
Contributor guide
Research direction
Start with curator/helpers/waiters.py and its restore_check function; compare the older 5.x implementation in curator/utils.py. Trace the _recovery polling and verify the restored indices through cluster health, using allocation explain details for red health. Done means a failed or missing shard cannot be reported as a successful restore.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, python
- Domain
- api, backend, cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100