[NIDX-02A] Encode Property documents as native ICE v3 segment bytes
- Dominant language
- Java
- Stars
- 25k
- Forks
- 6.6k
- Avg merge
- 10h 5m
- Merged PRs (30d)
- 16
Description
Parent: #14007
Blocked by: none — #14002 is closed and the native reader is on `main` at `31b32ca2`.
## Why this is a leaf and not a format-only ticket
The workstream forbids format-only encoder tickets because such a ticket has no consumer: nothing exercises the output, so RED/GREEN proves only self-consistency. This leaf has two consumers, both already on `main` and both already required by the NIDX-02 end state:
1. **The merged native reader.** `nativeice.Open(path string)` reads a shard directory directly and the package imports no retired index dependency. It already serves `pkg/index/inverted/repair_page.go` and `pkg/index/inverted/inverted.go`.
2. **The pinned compatibility reader**, per the workstream's requirement that native output be accepted by the pinned rollback binary.
This is the shape NIDX-01 used, run in the other direction: there an independent oracle produced fixture bytes and the native reader consumed them; here the native encoder produces and two independent readers verify.
No production write path changes in this leaf. `banyand/property/db/shard.go` is untouched.
## Boundary
One exported encoder in `pkg/index/inverted/internal/nativeice` that writes one committed ICE v3 generation — segment plus snapshot — for Property's current document mapping: `_id`, `_entity_id`, `_group`, `_source`, `_timestamp`, `_deleted`, `_sha_value`, and tag fields. Repeated stored values are preserved. CRC32 fields are present but never calculated or validated.
**Size bound, and it is the point of this leaf:** encode exactly what the existing reader parses and nothing more. This leaf does not implement the `segment.Segment` or `segment.Merger` contracts, does not register a plugin, and does not touch the lifecycle manager — `nativeice.Open` reads files, not interfaces, so none of that is reachable from here.
## Required behavior
- A generation written by the encoder is opened by `nativeice.Open` and yields the same visible document count, stored-document walk, exact-term selection, deletion masks, and ascending repair search-after order as the checked-in compatibility fixture for the same input documents.
- Deleted documents are absent from every supported query and scan.
- An injected failure before publication leaves no partially visible generation; `Open` selects the prior complete generation or reports the directory empty.
- The pinned compatibility reader opens the output for query and restart.
## RED
```
go test ./pkg/index/inverted/internal/nativeice/ -run TestNativeEncodeRoundTripsThroughTheMergedReader -count=1
```
Fails on current `main`: the package exposes no encoder or writer symbol at all, so this is a compile/contract RED, not a same-output mismatch.
## E2E
```
go test ./pkg/index/inverted/ -run TestNativeEncodedGenerationMatchesCompatibilityFixture -count=1
```
Encode the fixture's input documents, open the result with `nativeice.Open`, and assert row-for-row equality against `pkg/index/inverted/testdata/nidx02a`, whose bytes come from the compatibility writer at an immutable revision or content hash — never from the encoder under test.
## Focused suites
```
go test ./pkg/index/inverted/internal/nativeice/ ./pkg/index/inverted/
```
## Non-scope
Plugin adapter over the encoder, plugin registration, `newShard` cutover, Property-wide rollout selection, rollback drain, merge, expiry, GC, backup, range and boolean query, and explicit sort. Those are later leaves. Do not create a snapshot-only, merge-only, or GC-only ticket from any of them.
## Lexical gate
The repository delta must add **zero new case-insensitive retired-dependency tokens** and zero matching tracked paths, across imports and aliases, names, filenames, strings, comments, tests, fixtures, scripts, configuration, and generated assets. This leaf is reachable without any: every type it needs is available through the existing neutral `segment` alias, and it never references the aliased index package. Compatibility evidence uses neutral `legacy oracle` or `compatibility writer` labels plus an immutable revision or content hash.
Contributor guide
Research direction
Start with pkg/index/inverted/internal/nativeice and the existing nativeice.Open reader, then run TestNativeEncodeRoundTripsThroughTheMergedReader to inspect the missing encoder contract. Compare the encoded generation with pkg/index/inverted/testdata/nidx02a and run TestNativeEncodedGenerationMatchesCompatibilityFixture. Done means both readers accept the output and queries, scans, deletions, ordering, and failure publication behavior match the stated requirements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100