[NIDX-04 workstream] Replace the Stream element index
- Dominant language
- Java
- Stars
- 25k
- Forks
- 6.6k
- Avg merge
- 10h 5m
- Merged PRs (30d)
- 16
Description
# [NIDX-04 workstream] Replace the Stream element index
Parent: #13990
Blocked by: #14003
> **Tracking parent — do not apply `Backlog`.** Split this workstream only after NIDX-03 closes, using the actual native writer, reader, posting, and receive seams then present on `main`.
## End-state boundary
`banyand/stream.newElementIndex` selects the native `idx` batch writer, filter executor, posting collector, sorter, snapshot path, restart path, and external receiver.
## Required behavior
- Committed batches become visible at the documented near-real-time boundary.
- Exact, existence, range, AND/OR/NOT, HAVING, IN, prefix, wildcard, and MATCH return fixed document-ID sets.
- Keyword, simple, standard, and URL analysis produce fixed terms; MATCH AND/OR overrides remain distinct.
- Document-ID/timestamp postings and ascending/descending doc-value sort are deterministic.
- Legacy/native snapshots and raw segments are cross-readable; interrupted receive never publishes partial data.
- Scoring, phrase/fuzzy/geo, highlighting, facets, explanations, collectors, and plugins remain absent.
## Just-in-time decomposition gate
After #14003 closes, file ordered leaves around live Stream callers and one observable feature each. Every leaf needs a pre-agreed seam, literal terms/document IDs, a RED command, an e2e command, resource/concurrency bounds, and same-file rollback. Do not merge analyzers, filters, postings, sorters, or receiver code without activating a production path. Apply `Backlog` to one unblocked leaf only.
## Completion criteria
All future leaves merge; the complete `newElementIndex` role is native; fixture, NRT, analyzer, posting, sort, restart, receive, race, and compatibility suites pass; and this workstream closes before NIDX-05 decomposition.
## Lexical non-regression
Repository changes for every implementation leaf before the final removal may delete existing references but must add **zero new case-insensitive `bluge` tokens** and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutral `legacy oracle` or `compatibility writer` labels plus an immutable revision or content hash rather than adding a retired module name.
This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.
## Design
[BDB-NIDX-SPEC-001 revision 0.2 — NIDX-04 behavior](https://github.com/apache/skywalking-banyandb/blob/main/docs/design/archive/0.12.0/native-inverted-index/delivery-review.html#nidx-04)
Contributor guide
Research direction
Read the BDB-NIDX-SPEC-001 revision 0.2 design and inspect the banyand/stream.newElementIndex entry point after #14003 closes. The issue requires decomposition into ordered, unblocked leaves with named seams and RED/e2e commands; completion means the native role and all listed fixture, NRT, analyzer, posting, sort, restart, receive, race, and compatibility suites pass.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases, search
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100