apache / apache/skywalking

[NIDX-02B] Drive the native encoder through the segment plugin contract

Open
#14,074 0 comments 0 reactions 0 assignees View on GitHub
bydb file compatible change database feature
Dominant language
Java
Stars
25k
Forks
6.6k
Avg merge
10h 5m
Merged PRs (30d)
16

Description

Parent: #14007
Blocked by: #14073 (NIDX-02A). An open pull request does not unblock this; 02A must merge first.

> **Provisional shape.** This leaf is the weakest of the NIDX-02 ladder against the workstream's "no layer-shaped tickets" rule: its consumer is the index library's own lifecycle rather than a live Property caller. It is filed separately so the encoder (#14073) and the production cutover (NIDX-02C) each stay reviewable, and so the cutover's dependency-policy question does not block the encoder. **If maintainers judge it not vertical, fold it into NIDX-02C and close this as superseded** — that is a cheaper outcome than an oversized cutover leaf, and the decision should be made before this becomes the oldest unblocked leaf.

## Boundary

Wrap #14073's encoder in the `SegmentPlugin` contract so the existing lifecycle manager can drive it: `New(results []segment.Document, normCalc func(string, int) float32) (segment.Segment, uint64, error)`, `Load(*segment.Data) (segment.Segment, error)`, and `Merge([]segment.Segment, []*roaring.Bitmap, int) segment.Merger`.

The returned value must satisfy the full `segment.Segment` (14 methods) and `segment.Merger` (5 methods) contracts, not merely serialize bytes. Every type is reachable through the existing neutral `segment` alias.

**Not registered in this leaf.** No `WithSegmentPlugin` call, no change to `inverted.NewStore` configuration, no production behavior change. The plugin is constructed and driven only from tests.

## Delegation question this leaf must answer, not assume

The cheaper implementation wraps encoded bytes with `segment.NewDataBytes` and delegates `Load` to the pinned loader, which would give the full interface for free. That delegation reaches the retired dependency's loader and cannot be written without a new token in tracked source. **Do not promise delegation as available.** Either a pre-approved neutral bridge exists by the time this leaf starts, or the adapter implements the contracts natively. Resolve this before the first commit and record the answer in the PR description.

## Required behavior

- The lifecycle manager persists a segment produced by `New` through the normal `Segment.WriteTo` path, and `Load` reopens the persisted bytes.
- `Merge` produces a segment whose visible documents are the union of its inputs minus the supplied deletion bitmaps.
- Output remains readable by `nativeice.Open` and by the pinned compatibility reader, exactly as in #14073.

## RED / E2E / suites

To be sharpened against 02A's landed API when this becomes the oldest unblocked leaf. The RED anchors on the lifecycle manager driving the plugin — failing on `main` because no plugin implementation exists — and the e2e on a persist/reopen/merge cycle whose output the merged reader still accepts.

## Lexical gate

Zero new case-insensitive retired-dependency tokens or matching tracked paths. The `segment` alias is neutral and sufficient for the contracts; the aliased index package is not referenced from this leaf.

Contributor guide

Open the contributing guide

Research direction

Wait for #14073 to land, then read its encoder API alongside the existing SegmentPlugin lifecycle and the New, Load, Merge, Segment.WriteTo, and nativeice.Open entry points. Resolve whether a neutral bridge is approved before implementing; done means lifecycle persist/reopen and deletion-aware merge tests pass, both readers accept the output, and the lexical gate remains clean.

Written by the indexing model from the issue text.

Assessment

Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.