elastic / elastic/curator

restore_check reports success when failed shards disappear from _recovery (silent false success)

Open
#1,797 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.1k
Forks
627
PR merge metrics
No merged PRs in 30d

Description

**Expected Behavior**:
When a snapshot restore fails (a shard cannot be recovered, e.g. due to `IOException: No space left on device`), Curator's restore action should report a failure — or at least not report success.

**Actual Behavior**:
`restore_check` decides the restore is complete by polling `client.indices.recovery()` and returning True once every shard present in the response reports `stage: DONE`:
- master: `curator/helpers/waiters.py`, `restore_check` (~lines 141-189)(https://github.com/elastic/curator/blob/master/curator/helpers/waiters.py#L141-L189)
- older 5.x: `curator/utils.py`, lines 1744-1756 (https://github.com/elastic/curator/blob/v5.8.4/curator/utils.py)

The **trap**: if all the shards in the response show DONE, that does not automatically mean the restore finished successfully. Why? Because shards that fail disappear from the response (the shard is deallocated — UNASSIGNED/ALLOCATION_FAILED — and drops out of the /_recovery response entirely). So if a shard failed, you will not get that information from this endpoint. It will tell you that all the remaining shards are DONE, but that does not mean the restore succeeded.

## Steps to Reproduce
1. Snapshot indices larger than the free disk space on the target node
(in our case ~82 GB snapshot vs ~80 GB free).
2. Restore via Curator's restore action with `wait_for_completion: True`
(Curator-level waiting, i.e. the `restore_check` polling loop).
3. One shard hits `IOException: No space left on device`; Elasticsearch
retries allocation, gives up, and the shard leaves `_recovery`.
4. `restore_check` will see only the remaining shards, all with `stage: DONE`,
and return True — Curator reports success while `GET /_cluster/health` is red.
Note: we hit this bug in our own application, which uses the same
`_recovery`-based check; the Curator behavior is established by code
inspection of `restore_check`, not by a Curator run.

## Suggested Fix
After the all-DONE condition, verify the outcome before declaring success:
`GET /_cluster/health` for the restored indices (green/yellow → success),
and on `red` inspect `GET /_cluster/allocation/explain` for the unassigned
primary (`can_allocate: no` → fail with the reason from
`unassigned_info.details`).

## Context
- Same trap reported against the Elasticsearch docs:
elastic/elasticsearch#157608
- Kibana's Snapshot & Restore UI has the same issue: elastic/kibana#63154
- Versions: Curator behavior verified in master and v5.8.4;
Elasticsearch 7.17.27; RHEL 9.

Contributor guide

Open the contributing guide

Research direction

Start with curator/helpers/waiters.py and its restore_check function; compare the older 5.x implementation in curator/utils.py. Trace the _recovery polling and verify the restored indices through cluster health, using allocation explain details for red health. Done means a failed or missing shard cannot be reported as a successful restore.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, python
Domain
api, backend, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.