[Feature] Replace Bluge with a native ICE-compatible inverted index
- Dominant language
- Java
- Stars
- 25k
- Forks
- 6.6k
- Avg merge
- 10h 5m
- Merged PRs (30d)
- 16
Description
## Summary
Replace runtime Bluge/ICE with a minimal BanyanDB-owned inverted index while preserving ICE v3 and snapshot v3. BanyanDB owns index behavior; this is not a new-format migration.
The behavioral and compatibility authority remains [BDB-NIDX-SPEC-001 revision 0.2](https://github.com/apache/skywalking-banyandb/tree/main/docs/design/archive/0.12.0/native-inverted-index), introduced by [skywalking-banyandb#1306](https://github.com/apache/skywalking-banyandb/pull/1306) and refined by [skywalking-banyandb#1307](https://github.com/apache/skywalking-banyandb/pull/1307). The tracker hierarchy below refines implementation granularity so each automated TDD run owns one observable seam and one mergeable live-caller replacement.
## Boundary
Public `pkg/index/` contracts remain stable for callers. Native implementation types are internal; no Bluge, ICE, or `bluge_segment_api` type may escape. The five top-level NIDX issues are **tracking workstreams**, not implementation candidates:
1. bounded direct read-only operations;
2. Property shard index;
3. per-segment series `sidx`;
4. Stream element `idx`; and
5. remaining administration plus dependency removal.
## TDD execution model
The TDD workflow selects the lowest-number open issue carrying both `database` and `Backlog`. Therefore:
- never apply `Backlog` to this umbrella or a workstream parent;
- only the oldest unblocked **leaf** may carry `Backlog`;
- a successor is not unblocked until its predecessor PR merges;
- create downstream leaves just in time, after their blocker has taught us the real seam; and
- every leaf must switch a named production/CLI caller in its merge—no fixture-only, parser-only, codec-only, shadow-only, or test-only leaf.
```mermaid
graph LR
A["NIDX-01A
baseline count"] --> B["NIDX-01B
generations + deletions"]
B --> C["NIDX-01C
stored migration walks"]
C --> D["NIDX-01D
exact schema walks"]
D --> E["NIDX-01E
repair search-after"]
E --> P["decompose NIDX-02
from merged seams"]
P --> S["decompose NIDX-03"] --> T["decompose NIDX-04"] --> R["decompose NIDX-05"]
```
Pre-filing all downstream leaves would be horizontal slicing: their contracts would describe interfaces that do not exist yet and would be stale when selected. File them only when the preceding workstream closes; their issue numbers will then naturally follow dependency order.
## Current executable leaves: NIDX-01
1. [#14008 — baseline single-segment `ReadOnlyDocCount`](https://github.com/apache/skywalking/issues/14008)
2. [#14009 — multi-segment generations and deletion masks](https://github.com/apache/skywalking/issues/14009), blocked by #14008
3. [#14010 — match-all stored walks for migration sources](https://github.com/apache/skywalking/issues/14010), blocked by #14009
4. [#14011 — exact-term/OR schema walks](https://github.com/apache/skywalking/issues/14011), blocked by #14010
5. [#14012 — Property repair tuple search-after](https://github.com/apache/skywalking/issues/14012), blocked by #14011
Each leaf owns checked-in compatibility-writer fixture bytes, literal expected values, one pre-agreed behavioral seam, RED and e2e commands, focused race/package suites, immutable-file proof, typed bounded corruption behavior, and explicit rollback.
## Compatibility and durability contract
- Native readers open supported historical ICE v3/snapshot directories and return canonical logical results.
- During the retirement window, the pinned legacy oracle opens native output and can query, merge, restart, and externally receive it where applicable.
- No new disk version, opening-time migration, replay, or rewrite is allowed merely to remove dependencies.
- Required CRC32 fields remain layout bytes only; native code never calculates, validates, caches, or uses them.
- Publication is atomic; restart selects a complete preceding or new generation, never a partial snapshot.
- Malformed input returns bounded typed errors without panic, hang, leak, or unbounded allocation.
- Native index code uses BanyanDB database ownership for writer exclusion; it creates and requires no index-local PID or lock file.
- Read-only operations create no runtime file and do not change bytes, mtimes, or directory entries.
## Required final product surface
- **Bounded reads:** committed generation, visible count, match-all stored walk, exact term/OR, deletion masks, and Property repair search-after.
- **Property:** upsert/delete, stored fields, required filters/sort, callbacks, snapshots, recovery, expiry merge, GC, repair, and backup.
- **Series:** field-aware mutation/projection, identity matching/dictionaries, filters/sort, index-mode Measure, snapshots, cache reset, and mixed-version receive.
- **Stream elements:** batch/NRT, required filters/analyzers, paired postings, doc-value sort, snapshot/restart/receive.
- **Administration:** verify, rebuild, union/dedup, reconstruction, migration output, dump, and final dependency removal.
BM25, boosts, phrase/fuzzy/geo, highlighting, facets, explanations, generic collectors, and plugin compatibility remain out of scope.
## Lexical non-regression
Repository changes for every implementation leaf before the final removal may delete existing references but must add **zero new case-insensitive `bluge` tokens** and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutral `legacy oracle` or `compatibility writer` labels plus an immutable revision or content hash rather than adding a retired module name.
This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.
## Completion criteria
- Every leaf is merged and its named caller is native; no dormant native implementation lands.
- Product, historical-fixture, two-binary, crash, corruption, race, resource, and performance gates pass for the applicable workstream.
- Operational metrics, mismatch artifacts, recovery guidance, and rollback instructions are exercised.
- NIDX-01 through NIDX-04 add no new case-insensitive `bluge` token or matching tracked path in active repository surfaces.
- The last NIDX-05 removal leaf deletes every pre-existing case-insensitive `bluge` token and matching tracked path outside archived design/research evidence and Git history; no alias, filename, runtime name, fixture metadata, or generated license remains.
- Final dependency scans find no retired runtime or CLI edge; compatibility fixtures use neutral oracle identity and immutable hashes.
## Top-level workstreams
1. [NIDX-01 — bounded read-only access](https://github.com/apache/skywalking/issues/14002), with five ordered TDD leaves above
2. [NIDX-02 — Property shard index](https://github.com/apache/skywalking/issues/14007), decompose after NIDX-01 closes
3. [NIDX-03 — per-segment series `sidx`](https://github.com/apache/skywalking/issues/14003), decompose after NIDX-02 closes
4. [NIDX-04 — Stream element `idx`](https://github.com/apache/skywalking/issues/14004), decompose after NIDX-03 closes
5. [NIDX-05 — administration and dependency removal](https://github.com/apache/skywalking/issues/14005), decompose after NIDX-01 through NIDX-04 close and retirement is approved
Contributor guide
Research direction
This is an umbrella tracker, not an implementation candidate. Start with the ordered leaf issue #14008, then follow #14009 through #14012 as blockers clear. The overall work is done when all named callers use the native index, compatibility and recovery gates pass, and the dependency-removal criteria are satisfied.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend, database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100