Support writes with previous major lucene versions
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Allow write capabilities with previous major bwc lucene versions to suport rolling upgrades[2] on OpenSearch.
### Background
Customers using OpenSearch performs upgrades to move to latest version of OpenSearch software. One of the possible upgrades is rolling upgrade[2] where each node is upgraded one at a time. This upgrade process result in intermediate state where few nodes are on latest OpenSearch version while others are still running on older version, creating a state of mixed version cluster. This state does not work well for segment replication[1] enabled indices because the primary shard copies over the segment files onto replica shard copies. This works fine when all nodes are running on same version but during upgrades it is possible that replica shard copy be running on a older version node and thus, does not understand the segment files written with newer codec on primary shard. This results in replica shard failures and impacts search availability.
### Solution attempted and issue
OpenSearch attempted to solve mixed cluster issue by updating primary shard copies to keep using older codec until all replica copies are on latest software. This resulted in segment files written with older codec which replica shard can read. This works for upgrades where there is minor Lucene version bump but not when there is major Lucene bump. We identified from [4] and [manual test](https://github.com/opensearch-project/OpenSearch/issues/3881#issuecomment-1581689082) that Lucene moves older codecs into bwc-codecs and only allow reads with all previous major versions. Thus, there is no write compatibility with previous major lucene versions and solution attempted on OpenSearch will not work for major Lucene version upgrades.
### References
[1] [Segment replication in OpenSearch ](https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/segment-replication/index/)
[2] [Rolling upgrades in OpenSearch ](https://opensearch.org/docs/latest/install-and-configure/upgrade-opensearch/rolling-upgrade/)
[3] [OpenSearch engine issue](https://github.com/opensearch-project/OpenSearch/issues/3881)
[4] [Backward codecs in Lucene ](https://github.com/apache/lucene/tree/main/lucene/backward-codecs)
Contributor guide
Research direction
Start with Lucene's lucene/backward-codecs directory and the linked OpenSearch engine issue 3881. Review how backward codecs currently support reads and compare that with the segment-replication rolling-upgrade scenario described in the issue. Done means a design and implementation that permits writes compatible with previous major Lucene versions without causing replica failures during mixed-version upgrades.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100