[NIDX-02 workstream] Replace the Property shard index
- Dominant language
- Java
- Stars
- 25k
- Forks
- 6.6k
- Avg merge
- 10h 5m
- Merged PRs (30d)
- 16
Description
# [NIDX-02 workstream] Replace the Property shard index
Parent: #13990
Blocked by: #14002
> **Tracking parent — do not apply `Backlog`.** Decompose this workstream into ordered TDD leaves only after every NIDX-01 leaf has merged and the actual native reader seam is available on `main`.
## End-state boundary
`banyand/property/db.newShard` selects the native implementation for existing and new Property shards. The completed workstream owns compatible Property writes, exact/boolean/range queries, explicit sort, callbacks, snapshot publication/recovery, merge-time expiry, reference-safe GC, repair integration, backup, and same-file rollback.
The retained legacy Property constructor is an explicit rollback choice during the compatibility window, not a second active writer.
## Cutover and writer ownership
The implementation choice is Property-database-wide during NIDX-02, not inferred independently for each shard.
- The Property database establishes exclusive writer ownership before scanning its directory or opening any shard store.
- While it owns the database, startup resolves exactly one implementation from an explicit Property-wide rollout selection; `property/db.newShard` uses that selection for every existing and newly created shard.
- A process that cannot establish ownership fails before either implementation opens a shard writer.
- The native index neither creates nor requires an index-local PID file, lock file, or writer-origin marker. Such runtime files are outside ICE/snapshot compatibility.
- Selection is not inferred from ICE segments, snapshot manifests, CRC32 values, or other writer-origin heuristics; those files remain implementation-neutral during the same-file rollback window.
- Rollback stops admission, drains durable callbacks, closes every active shard writer, changes the Property-wide selection, and then reopens the same files with the retained legacy constructor.
- Simultaneous native and legacy writers for the Property database are prohibited.
## Required behavior
- Legacy-created shards open without rewrite and retain documents, repeated stored values, deletions, and explicit order.
- Replace/upsert leaves one visible latest document; delete removes it from every supported query and scan.
- Native output is ICE v3/snapshot v3 accepted by the pinned rollback binary for query, mutation, merge, and restart.
- Publication and callbacks are durable; every crash cut selects a complete old or new generation.
- Merge, expiry, and GC retain only data reachable from protected snapshots.
- CRC32 fields remain present but are never calculated or validated.
## Just-in-time decomposition gate
When #14002 closes:
1. inspect the merged native reader and measured NIDX-01 run reports;
2. identify the smallest live Property caller for each new writer/lifecycle capability;
3. file leaf issues in dependency order so issue numbers ascend;
4. give each leaf one pre-agreed seam, literal fixture results, one RED command, one e2e command, and focused package suites; and
5. apply `Backlog` only to the first unblocked leaf.
Do not pre-file format-only writer, snapshot, merge, or GC tickets. Each leaf must activate a live Property behavior in its merge.
## Completion criteria
All future leaves merge; the complete `property/db.newShard` role is native; the two-binary compatibility and crash matrices pass; and the workstream closes before NIDX-03 decomposition begins.
## Lexical non-regression
Repository changes for every implementation leaf before the final removal may delete existing references but must add **zero new case-insensitive `bluge` tokens** and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutral `legacy oracle` or `compatibility writer` labels plus an immutable revision or content hash rather than adding a retired module name.
This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.
## Design
[BDB-NIDX-SPEC-001 revision 0.2 — NIDX-02 behavior](https://github.com/apache/skywalking-banyandb/blob/main/docs/design/archive/0.12.0/native-inverted-index/delivery-review.html#nidx-02)
Contributor guide
Research direction
Start by reviewing the merged native reader and measured NIDX-01 reports after #14002 closes. Use banyand/property/db.newShard and the stated completion criteria to identify ordered leaf issues; this workstream is done only when the native role, compatibility and crash matrices, and dependent leaves are complete.
Written by the indexing model from the issue text.
Assessment
- Domain
- database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100