apache / apache/skywalking

[NIDX-03 workstream] Replace the per-segment series index

Open
#14,003 0 comments 0 reactions 0 assignees View on GitHub
bydb file compatible change database
Dominant language
Java
Stars
25k
Forks
6.6k
Avg merge
10h 5m
Merged PRs (30d)
16

Description

# [NIDX-03 workstream] Replace the per-segment series index

Parent: #13990
Blocked by: #14007

> **Tracking parent — do not apply `Backlog`.** Split this workstream just in time after NIDX-02 closes; downstream RED contracts must be based on the writer and reader seams actually merged on `main`.

## End-state boundary

`banyand/internal/storage.newSeriesIndex` selects native `sidx` for Measure, Stream, and Trace. The completed workstream owns field-set-aware insert/update/delete, projection, exact/prefix/wildcard identity, field dictionaries, composed filters, deterministic series/index/time sort, index-mode Measure, statistics, snapshots, cache reset, and mixed-version external segment receive.

## Required behavior

- Fixed product fixtures return independently declared series IDs, fields, timestamps, and versions.
- Native and pinned legacy nodes exchange raw segments in both directions without replay or conversion.
- Interrupted, invalid, and duplicate external transfers never become partially visible.
- Restart selects a complete committed generation.
- CRC32 fields remain layout-only and are never calculated or validated.

## Just-in-time decomposition gate

After #14007 closes, use its measured boundaries and run reports to file ordered live-caller leaves. Each leaf adds one observable series capability and activates it through a named Measure/Stream/Trace path; no `sidx` core, codec, query helper, or receiver may merge solely for future use. Apply `Backlog` to one unblocked leaf only.

## Completion criteria

All future leaves merge; the complete `newSeriesIndex` role and replication path are native; compatibility, crash, race, and product suites pass; and this workstream closes before NIDX-04 decomposition.

## Lexical non-regression

Repository changes for every implementation leaf before the final removal may delete existing references but must add **zero new case-insensitive `bluge` tokens** and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutral `legacy oracle` or `compatibility writer` labels plus an immutable revision or content hash rather than adding a retired module name.

This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.

## Design

[BDB-NIDX-SPEC-001 revision 0.2 — NIDX-03 behavior](https://github.com/apache/skywalking-banyandb/blob/main/docs/design/archive/0.12.0/native-inverted-index/delivery-review.html#nidx-03)

Contributor guide

Open the contributing guide

Research direction

The named entry point is banyand/internal/storage.newSeriesIndex; first read BDB-NIDX-SPEC-001 revision 0.2 and wait for #14007's measured boundaries and reports. Use those boundaries to identify ordered live-caller leaves and their named Measure/Stream/Trace paths. Done means the native role and replication path meet the listed compatibility, crash, race, product, restart, transfer, and lexical gates.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.