apache / apache/skywalking

[NIDX-02A] Encode Property documents as native ICE v3 segment bytes

Open
#14,073 0 comments 0 reactions 0 assignees View on GitHub
accepted bydb file compatible change database feature
Dominant language
Java
Stars
25k
Forks
6.6k
Avg merge
10h 5m
Merged PRs (30d)
16

Description

Parent: #14007
Blocked by: none — #14002 is closed and the native reader is on `main` at `31b32ca2`.

## Why this is a leaf and not a format-only ticket

The workstream forbids format-only encoder tickets because such a ticket has no consumer: nothing exercises the output, so RED/GREEN proves only self-consistency. This leaf has two consumers, both already on `main` and both already required by the NIDX-02 end state:

1. **The merged native reader.** `nativeice.Open(path string)` reads a shard directory directly and the package imports no retired index dependency. It already serves `pkg/index/inverted/repair_page.go` and `pkg/index/inverted/inverted.go`.
2. **The pinned compatibility reader**, per the workstream's requirement that native output be accepted by the pinned rollback binary.

This is the shape NIDX-01 used, run in the other direction: there an independent oracle produced fixture bytes and the native reader consumed them; here the native encoder produces and two independent readers verify.

No production write path changes in this leaf. `banyand/property/db/shard.go` is untouched.

## Boundary

One exported encoder in `pkg/index/inverted/internal/nativeice` that writes one committed ICE v3 generation — segment plus snapshot — for Property's current document mapping: `_id`, `_entity_id`, `_group`, `_source`, `_timestamp`, `_deleted`, `_sha_value`, and tag fields. Repeated stored values are preserved. CRC32 fields are present but never calculated or validated.

**Size bound, and it is the point of this leaf:** encode exactly what the existing reader parses and nothing more. This leaf does not implement the `segment.Segment` or `segment.Merger` contracts, does not register a plugin, and does not touch the lifecycle manager — `nativeice.Open` reads files, not interfaces, so none of that is reachable from here.

## Required behavior

- A generation written by the encoder is opened by `nativeice.Open` and yields the same visible document count, stored-document walk, exact-term selection, deletion masks, and ascending repair search-after order as the checked-in compatibility fixture for the same input documents.
- Deleted documents are absent from every supported query and scan.
- An injected failure before publication leaves no partially visible generation; `Open` selects the prior complete generation or reports the directory empty.
- The pinned compatibility reader opens the output for query and restart.

## RED

```
go test ./pkg/index/inverted/internal/nativeice/ -run TestNativeEncodeRoundTripsThroughTheMergedReader -count=1
```

Fails on current `main`: the package exposes no encoder or writer symbol at all, so this is a compile/contract RED, not a same-output mismatch.

## E2E

```
go test ./pkg/index/inverted/ -run TestNativeEncodedGenerationMatchesCompatibilityFixture -count=1
```

Encode the fixture's input documents, open the result with `nativeice.Open`, and assert row-for-row equality against `pkg/index/inverted/testdata/nidx02a`, whose bytes come from the compatibility writer at an immutable revision or content hash — never from the encoder under test.

## Focused suites

```
go test ./pkg/index/inverted/internal/nativeice/ ./pkg/index/inverted/
```

## Non-scope

Plugin adapter over the encoder, plugin registration, `newShard` cutover, Property-wide rollout selection, rollback drain, merge, expiry, GC, backup, range and boolean query, and explicit sort. Those are later leaves. Do not create a snapshot-only, merge-only, or GC-only ticket from any of them.

## Lexical gate

The repository delta must add **zero new case-insensitive retired-dependency tokens** and zero matching tracked paths, across imports and aliases, names, filenames, strings, comments, tests, fixtures, scripts, configuration, and generated assets. This leaf is reachable without any: every type it needs is available through the existing neutral `segment` alias, and it never references the aliased index package. Compatibility evidence uses neutral `legacy oracle` or `compatibility writer` labels plus an immutable revision or content hash.

Contributor guide

Open the contributing guide

Research direction

Start with pkg/index/inverted/internal/nativeice and the existing nativeice.Open reader, then run TestNativeEncodeRoundTripsThroughTheMergedReader to inspect the missing encoder contract. Compare the encoded generation with pkg/index/inverted/testdata/nidx02a and run TestNativeEncodedGenerationMatchesCompatibilityFixture. Done means both readers accept the output and queries, scans, deletions, ordering, and failure publication behavior match the stated requirements.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.