[NIDX-03 workstream] Replace the per-segment series index
- Dominant language
- Java
- Stars
- 25k
- Forks
- 6.6k
- Avg merge
- 10h 5m
- Merged PRs (30d)
- 16
Description
# [NIDX-03 workstream] Replace the per-segment series index
Parent: #13990
Blocked by: #14007
> **Tracking parent — do not apply `Backlog`.** Split this workstream just in time after NIDX-02 closes; downstream RED contracts must be based on the writer and reader seams actually merged on `main`.
## End-state boundary
`banyand/internal/storage.newSeriesIndex` selects native `sidx` for Measure, Stream, and Trace. The completed workstream owns field-set-aware insert/update/delete, projection, exact/prefix/wildcard identity, field dictionaries, composed filters, deterministic series/index/time sort, index-mode Measure, statistics, snapshots, cache reset, and mixed-version external segment receive.
## Required behavior
- Fixed product fixtures return independently declared series IDs, fields, timestamps, and versions.
- Native and pinned legacy nodes exchange raw segments in both directions without replay or conversion.
- Interrupted, invalid, and duplicate external transfers never become partially visible.
- Restart selects a complete committed generation.
- CRC32 fields remain layout-only and are never calculated or validated.
## Just-in-time decomposition gate
After #14007 closes, use its measured boundaries and run reports to file ordered live-caller leaves. Each leaf adds one observable series capability and activates it through a named Measure/Stream/Trace path; no `sidx` core, codec, query helper, or receiver may merge solely for future use. Apply `Backlog` to one unblocked leaf only.
## Completion criteria
All future leaves merge; the complete `newSeriesIndex` role and replication path are native; compatibility, crash, race, and product suites pass; and this workstream closes before NIDX-04 decomposition.
## Lexical non-regression
Repository changes for every implementation leaf before the final removal may delete existing references but must add **zero new case-insensitive `bluge` tokens** and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutral `legacy oracle` or `compatibility writer` labels plus an immutable revision or content hash rather than adding a retired module name.
This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.
## Design
[BDB-NIDX-SPEC-001 revision 0.2 — NIDX-03 behavior](https://github.com/apache/skywalking-banyandb/blob/main/docs/design/archive/0.12.0/native-inverted-index/delivery-review.html#nidx-03)
Contributor guide
Research direction
The named entry point is banyand/internal/storage.newSeriesIndex; first read BDB-NIDX-SPEC-001 revision 0.2 and wait for #14007's measured boundaries and reports. Use those boundaries to identify ordered live-caller leaves and their named Measure/Stream/Trace paths. Done means the native role and replication path meet the listed compatibility, crash, race, product, restart, transfer, and lexical gates.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100