apache / apache/skywalking

[NIDX-05 workstream] Replace remaining administration and remove Bluge

Open
#14,005 0 comments 0 reactions 0 assignees View on GitHub
bydb file compatible change database
Dominant language
Java
Stars
25k
Forks
6.6k
Avg merge
10h 5m
Merged PRs (30d)
16

Description

# [NIDX-05 workstream] Replace remaining administration and remove Bluge

Parent: #13990
Blocked by: #14002, #14007, #14003, and #14004

> **Tracking parent — do not apply `Backlog`.** Split this workstream only after NIDX-01 through NIDX-04 close. Dependency removal also requires explicit legacy-retirement approval.

## End-state boundary

Every remaining verify, rebuild, union/dedup, index-mode copy destination, migration output, dump, maintenance, and CLI path uses native APIs. Runtime and offline retired dependencies are removed. Compatibility evidence remains only as neutral, immutable oracle fixtures; it does not retain a direct library import, module name, or runtime fallback.

## Required behavior

- Verify reports fixed valid counts and deterministic artifacts for missing segments, corrupt offsets, duplicate identities, and logical mismatches.
- Rebuild, union/dedup, copy, and migration destinations publish exactly the declared logical records through crash-safe staging.
- Dump and remaining callers handle historical and native directories through bounded native APIs.
- The full two-binary matrix passes before dependency removal.
- Removal introduces no format version, opening rewrite, replay, CRC32 processing, or unsupported search feature.
- Writer exclusion is owned by BanyanDB's database lock; delete the obsolete index-local PID filename and every constant, filter, fixture, and test that names or depends on it.

## Just-in-time decomposition gate

After all four predecessor workstreams close, file ordered leaves for the live remaining commands observed at that point. Keep verify, rebuild/union, copy/migration, dump/maintenance, and final dependency removal separate when their seams differ. Every non-removal leaf must replace a named production or CLI caller. The removal leaf is last and blocked by its sibling merges plus retirement approval. It must remove the remaining keyword occurrences and paths rather than merely making them unreachable. Apply `Backlog` to one unblocked leaf only.

## Completion criteria

All future leaves merge; compatibility and crash matrices pass; and all of these final scans produce no output:

```bash
git grep -in bluge -- ':!docs/design/archive/**'
git ls-files | grep -i bluge
go list -deps ./... | grep -i bluge
```

The content scan covers the active repository surface, including Go source and tests, active documentation, modules, scripts/configuration, fixtures/provenance, messages, and generated license assets. The path scan covers every tracked filename. Archived design/research records and Git history are the only historical exemptions. No misspelling, split string, or neutral-looking alias may retain a runtime dependency. Once these gates pass, the umbrella may close.

## Final lexical removal

NIDX-05 is the contract phase of the expand–migrate–contract sequence. Earlier implementation leaves add zero new case-insensitive `bluge` tokens or matching tracked paths; the **last removal leaf deletes every pre-existing occurrence and matching path from the active repository surface**. This includes imports, aliases, identifiers, filenames, the obsolete runtime filename, strings/messages, tests, fixtures/provenance, modules, scripts, configuration, active docs, and generated assets.

Compatibility evidence must use neutral oracle labels plus immutable revision/content hashes. Archived design/research records and Git history remain as provenance because they document the retired dependency; no active code, test, tool, or generated output may require them.

## Design

[BDB-NIDX-SPEC-001 revision 0.2 — NIDX-05 behavior](https://github.com/apache/skywalking-banyandb/blob/main/docs/design/archive/0.12.0/native-inverted-index/delivery-review.html#nidx-05)

Contributor guide

Open the contributing guide

Research direction

Start with the archived BDB-NIDX-SPEC-001 design and wait for blocked issues #14002, #14007, #14003, and #14004 to close. Then inspect the live verify, rebuild/union, copy/migration, dump/maintenance, and CLI callers and split ordered leaves by their seams. Completion is shown by the compatibility and crash matrices plus the three specified grep/list scans producing no output.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, cli, databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.