HarperFast / HarperFast/harper

Qualify Harper full-text search at catalog scale for v5.3

Open
#2,512 0 comments 0 reactions 1 assignee Claimed by @kylebernhardy View on GitHub
area:performance area:search tests
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Outcome

Qualify the complete Harper feature using local native Tantivy indexes and explicit workload/recovery objectives.

## Acceptance criteria

- Define realistic product field sizes, total text bytes, query/filter distribution, QPS, update rate, hardware and rebuild-time targets.
- End-to-end search p99 below 50 ms for every accepted query class on the agreed workload; report recovery/cold-cache regimes separately.
- Compare Harper without fulltext, HNSW where applicable, one fulltext index, and multiple native fulltext indexes alongside normal primary-store traffic.
- Use standalone native-wrapper measurements as engine context; integrated overhead includes source writes/projection/replay/auth/retrieval.
- Exercise local/replicated writes, initial replica copy, eviction, replay, source restore, worker handoff/loss, rebuild, generation activation and close/drop.
- Successful source changes meet the agreed steady-state visibility objective, with backlog/drain, errors and overload reported.
- Measure disk/mmap memory, merge/rebuild headroom and contention with RocksDB on shared devices; avoid assumed fixed space multipliers.
- Preserve process-wide ceilings and query safety for fuzzy/prefix/highlighting.
- Pin and qualify the Fulltext artifact and Tantivy formats with the supported Harper version; no special fulltext/rocksdb-js ABI pair.
- Automate cross-repo CI fixtures and controlled release benchmarks; keep compatible immutable result history in GitHub.

## Design

[Native Tantivy storage and Harper derived indexes](https://github.com/HarperFast/fulltext/blob/codex/native-storage-design/docs/native-storage-integration.md). Harper owns source state and derived coordination; Tantivy uses native local files. This updates the design, not implementation status.

Comment generated by kAIle (GPT-6)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.