elastic / elastic/elasticsearch

Improve handling of external forces that change repo contents

Open
#82,102 3 comments 0 reactions 0 assignees View on GitHub
:Distributed/Snapshot/Restore >bug Team:Distributed
Dominant language
Java
Stars
77.9k
Forks
26.1k
PR merge metrics
PR metrics pending

Description

Today in a writeable repository we rely on the assumption that the only reason the repository contents might change is because we changed it, which lets the master maintain a cache of some of its contents. We only validate this cache when writing to the repository.

In practice it seems pretty common to register an empty repository and then restore its contents from a backup:

- [#293024](https://discuss.elastic.co/t/cant-move-large-es-snapshots-to-different-server/293024?u=davidturner)
- [#289226](https://discuss.elastic.co/t/synced-snapshots-in-s3-dont-show-up-as-available-in-the-repository/289226?u=davidturner)
- [#286372](https://discuss.elastic.co/t/restore-data-from-archive-snapshot/286372?u=davidturner)
- [#284292](https://discuss.elastic.co/t/restoring-snapshot-to-another-cluster/284292?u=davidturner)

In this workflow Elasticsearch does not write to the repository so the cached data is never invalidated, and therefore a listing of the repository contents remains empty even after the restore process is complete. This is surprising to users, and since Elasticsearch gives no indication why it's behaving in this surprising way I think this is a bug.

Elasticsearch should be able to detect unexpected `index-N` files when listing the repository contents after restoring the repo from a snapshot, and should treat it as an error if one is found and use the error response to guide users towards fixing their workflow (e.g. setting `readonly: true` or delaying registration until the restore is complete).

Checking for rogue `index-N` files each time we list the repository contents might be ok but might also be too frequent. Maybe we should only do this check if the last time we checked was more than N seconds ago. Or maybe only if the repo is currently empty. Or maybe only if we've not yet written to the repository. Opening this for discussion about possible alternatives.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.