HarperFast / HarperFast/harper

Epic: Integrate schema-native full-text search into Harper

Open
#2,509 0 comments 0 reactions 1 assignee Claimed by @kylebernhardy View on GitHub
area:search enhancement
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Outcome

Use the native @harperfast/fulltext wrapper as a first-class schema-declared derived index in Harper v5.3.

## Acceptance criteria

- Use native Tantivy files for each node's local index; source records remain in Harper's primary store.
- Reuse the shared lock-elected derived runtime for local and replicated changes, replay, backpressure, ownership and rebuild.
- Add @fullText following embed-field conventions, with weighted sources, English analysis, configurable positions/surface terms, and opt-in synonyms/highlighting.
- Compose full-text conditions and $score with existing Table.search and REST planning and authorization.
- Reuse valid index files on restart; replay exact local checkpoints. Fresh replicas and source restores rebuild locally before full-text queries become available.
- Coordinate generation identity/swap, eviction, worker loss, close/drop, restore invalidation, cleanup and status.
- Qualify native checkpoint publication and catalog-scale correctness/performance, including mixed normal Harper traffic.
- Keep RocksDB as the initial qualified source runtime; LMDB support is not implied by native index files.
- No hosted Directory, /harper or /rocks storage factory, private provider, or new rocksdb-js primitive is part of fulltext.

## Design

[Native Tantivy storage and Harper derived indexes](https://github.com/HarperFast/fulltext/blob/codex/native-storage-design/docs/native-storage-integration.md). Harper owns source state and derived coordination; Tantivy uses native local files. This updates the design, not implementation status.

Comment generated by kAIle (GPT-6)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.