apache / apache/skywalking

[NIDX-02 workstream] Replace the Property shard index

Open
#14,007 2 comments 0 reactions 0 assignees View on GitHub
bydb file compatible change database
Dominant language
Java
Stars
25k
Forks
6.6k
Avg merge
10h 5m
Merged PRs (30d)
16

Description

# [NIDX-02 workstream] Replace the Property shard index

Parent: #13990
Blocked by: #14002

> **Tracking parent — do not apply `Backlog`.** Decompose this workstream into ordered TDD leaves only after every NIDX-01 leaf has merged and the actual native reader seam is available on `main`.

## End-state boundary

`banyand/property/db.newShard` selects the native implementation for existing and new Property shards. The completed workstream owns compatible Property writes, exact/boolean/range queries, explicit sort, callbacks, snapshot publication/recovery, merge-time expiry, reference-safe GC, repair integration, backup, and same-file rollback.

The retained legacy Property constructor is an explicit rollback choice during the compatibility window, not a second active writer.

## Cutover and writer ownership

The implementation choice is Property-database-wide during NIDX-02, not inferred independently for each shard.

- The Property database establishes exclusive writer ownership before scanning its directory or opening any shard store.
- While it owns the database, startup resolves exactly one implementation from an explicit Property-wide rollout selection; `property/db.newShard` uses that selection for every existing and newly created shard.
- A process that cannot establish ownership fails before either implementation opens a shard writer.
- The native index neither creates nor requires an index-local PID file, lock file, or writer-origin marker. Such runtime files are outside ICE/snapshot compatibility.
- Selection is not inferred from ICE segments, snapshot manifests, CRC32 values, or other writer-origin heuristics; those files remain implementation-neutral during the same-file rollback window.
- Rollback stops admission, drains durable callbacks, closes every active shard writer, changes the Property-wide selection, and then reopens the same files with the retained legacy constructor.
- Simultaneous native and legacy writers for the Property database are prohibited.

## Required behavior

- Legacy-created shards open without rewrite and retain documents, repeated stored values, deletions, and explicit order.
- Replace/upsert leaves one visible latest document; delete removes it from every supported query and scan.
- Native output is ICE v3/snapshot v3 accepted by the pinned rollback binary for query, mutation, merge, and restart.
- Publication and callbacks are durable; every crash cut selects a complete old or new generation.
- Merge, expiry, and GC retain only data reachable from protected snapshots.
- CRC32 fields remain present but are never calculated or validated.

## Just-in-time decomposition gate

When #14002 closes:

1. inspect the merged native reader and measured NIDX-01 run reports;
2. identify the smallest live Property caller for each new writer/lifecycle capability;
3. file leaf issues in dependency order so issue numbers ascend;
4. give each leaf one pre-agreed seam, literal fixture results, one RED command, one e2e command, and focused package suites; and
5. apply `Backlog` only to the first unblocked leaf.

Do not pre-file format-only writer, snapshot, merge, or GC tickets. Each leaf must activate a live Property behavior in its merge.

## Completion criteria

All future leaves merge; the complete `property/db.newShard` role is native; the two-binary compatibility and crash matrices pass; and the workstream closes before NIDX-03 decomposition begins.

## Lexical non-regression

Repository changes for every implementation leaf before the final removal may delete existing references but must add **zero new case-insensitive `bluge` tokens** and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutral `legacy oracle` or `compatibility writer` labels plus an immutable revision or content hash rather than adding a retired module name.

This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.

## Design

[BDB-NIDX-SPEC-001 revision 0.2 — NIDX-02 behavior](https://github.com/apache/skywalking-banyandb/blob/main/docs/design/archive/0.12.0/native-inverted-index/delivery-review.html#nidx-02)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the merged native reader and measured NIDX-01 reports after #14002 closes. Use banyand/property/db.newShard and the stated completion criteria to identify ordered leaf issues; this workstream is done only when the native role, compatibility and crash matrices, and dependent leaves are complete.

Written by the indexing model from the issue text.

Assessment

Domain
database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.