opensearch-project / opensearch-project/OpenSearch
[Writable Warm] initialize a writeable warm index from a snapshot
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 13.7k
- Forks
- 3k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 108
Description
Is your feature request related to a problem? Please describe
This is an optimization related to the Writeable Warm feature.
When the writeable warm feature is introduced we will have the ability to create indices and then migrate them to a warm tier. A use case this does not cover is to create a warm index from a snapshot without having to go through expensive segment download/re-upload as a remote backed index.
Describe the solution you'd like
I think we can make this happen with a new RemoteDirectory implementation that conditionally fetches from a blob store wired to an existing snapshot or another with the remote store directory. This new dir could be injected into RemoteSegmentStoreDirectory as its data directory. All metadata continues to push to the remote store as normal it is only when fetching a file that we would interface with the original snapshot if necessary. In a way its similar to a searchable snapshot dir however those code paths would not be reusable with the incoming writeable warm CompositeDirectory implementation. That dir handles block fetch above RemoteSegmentStoreDirectory.
This would look something like below with new writes would push to the remote store as normal and reads flowing through FilteredRemoteDirectory.
A requirement here would be to enforce some level of deletion protection on the original snapshot for the lifetime of the index or at least until all segments from the original snapshot are merged away. We could do this with some new index level settings to validate at snapshot deletion time to ensure its not backing any existing index, similar to searchable snapshots.
Related component
Storage:Remote
Describe alternatives you've considered
Nothing - restore from snapshot as a hot index then migrate.
Migrate data off cluster and somehow wire it up when the dir initializes. This is risky because remote store paths are determined at index creation.
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the related Writeable Warm issue #12809 and the named RemoteDirectory and RemoteSegmentStoreDirectory components. Trace how snapshot-backed reads and remote-store writes currently work. Done would mean a warm index can initialize from a snapshot while preserving normal metadata writes and enforcing the required snapshot deletion protection, but the issue names no files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100