apache / apache/skywalking

[NIDX-04 workstream] Replace the Stream element index

Open
#14,004 0 comments 0 reactions 0 assignees View on GitHub
bydb file compatible change database
Dominant language
Java
Stars
25k
Forks
6.6k
Avg merge
10h 5m
Merged PRs (30d)
16

Description

# [NIDX-04 workstream] Replace the Stream element index

Parent: #13990
Blocked by: #14003

> **Tracking parent — do not apply `Backlog`.** Split this workstream only after NIDX-03 closes, using the actual native writer, reader, posting, and receive seams then present on `main`.

## End-state boundary

`banyand/stream.newElementIndex` selects the native `idx` batch writer, filter executor, posting collector, sorter, snapshot path, restart path, and external receiver.

## Required behavior

- Committed batches become visible at the documented near-real-time boundary.
- Exact, existence, range, AND/OR/NOT, HAVING, IN, prefix, wildcard, and MATCH return fixed document-ID sets.
- Keyword, simple, standard, and URL analysis produce fixed terms; MATCH AND/OR overrides remain distinct.
- Document-ID/timestamp postings and ascending/descending doc-value sort are deterministic.
- Legacy/native snapshots and raw segments are cross-readable; interrupted receive never publishes partial data.
- Scoring, phrase/fuzzy/geo, highlighting, facets, explanations, collectors, and plugins remain absent.

## Just-in-time decomposition gate

After #14003 closes, file ordered leaves around live Stream callers and one observable feature each. Every leaf needs a pre-agreed seam, literal terms/document IDs, a RED command, an e2e command, resource/concurrency bounds, and same-file rollback. Do not merge analyzers, filters, postings, sorters, or receiver code without activating a production path. Apply `Backlog` to one unblocked leaf only.

## Completion criteria

All future leaves merge; the complete `newElementIndex` role is native; fixture, NRT, analyzer, posting, sort, restart, receive, race, and compatibility suites pass; and this workstream closes before NIDX-05 decomposition.

## Lexical non-regression

Repository changes for every implementation leaf before the final removal may delete existing references but must add **zero new case-insensitive `bluge` tokens** and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutral `legacy oracle` or `compatibility writer` labels plus an immutable revision or content hash rather than adding a retired module name.

This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.

## Design

[BDB-NIDX-SPEC-001 revision 0.2 — NIDX-04 behavior](https://github.com/apache/skywalking-banyandb/blob/main/docs/design/archive/0.12.0/native-inverted-index/delivery-review.html#nidx-04)

Contributor guide

Open the contributing guide

Research direction

Read the BDB-NIDX-SPEC-001 revision 0.2 design and inspect the banyand/stream.newElementIndex entry point after #14003 closes. The issue requires decomposition into ordered, unblocked leaves with named seams and RED/e2e commands; completion means the native role and all listed fixture, NRT, analyzer, posting, sort, restart, receive, race, and compatibility suites pass.

Written by the indexing model from the issue text.

Assessment

Domain
databases, search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.