FilteringCodec, TeeCodec, TeeDirectory [LUCENE-2632]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
This issue adds two new Codec implementations:
- TeeCodec: there have been attempts in the past to implement parallel writing to multiple indexes so that they are all synchronized. This was however complicated due to the complexity of IndexWriter/SegmentMerger logic. The solution presented here offers a similar functionality but working on a different level - as the name suggests, the TeeCodec duplicates index data into multiple output Directories.
- TeeDirectory (used also in TeeCodec) is a simple abstraction to perform Directory operations on several directories in parallel (effectively mirroring their data). Optionally it's possible to specify a set of suffixes of files that should be mirrored so that non-matching files are skipped.
- FilteringCodec is related in a remote way to the ideas of index pruning presented in #2887 and the concept of tiered search. Since we can use TeeCodec to write to multiple output Directories in a synchronized way, we could also filter out or modify some of the data that is being written. The FilteringCodec provides this functionality, so that you can use like this:
```Java
IndexWriter --> TeeCodec
| |
| +--> StandardCodec --> Directory1
+--> FilteringCodec --> StandardCodec --> Directory2
```
The end result of this chain is two indexes that are kept in sync - one is the full regular index, and the other one is a filtered index.
---
Migrated from [LUCENE-2632](https://issues.apache.org/jira/browse/LUCENE-2632) by Andrzej Bialecki (@sigram), updated Sep 28 2012
Attachments: [LUCENE-2632.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2632/LUCENE-2632.patch) (versions: 5), [LUCENE-2632-filtering.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2632/LUCENE-2632-filtering.patch), [LUCENE-2632-trunk.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2632/LUCENE-2632-trunk.patch)
Contributor guide
Research direction
Start by reviewing the attached LUCENE-2632 patches and the existing IndexWriter, StandardCodec, and Directory abstractions. Trace how TeeCodec, TeeDirectory, and FilteringCodec are expected to compose, then verify that mirrored directories remain synchronized while suffix filtering produces the intended secondary index.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100