HarperFast / HarperFast/harper
Implement full-text generation lifecycle, rebuild, eviction, backup, and status
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Outcome
Connect native Tantivy files to Harper's shared derived runtime and database/index lifecycle.
## Acceptance criteria
- Use the existing elected transaction-log runner, not direct per-worker aftercommit delivery; resolve current committed primary projections.
- Adapt native open/publish/close to FullTextDerivedIndexBackend and store the exact offered cursor inside the Tantivy commit payload.
- Verify native queue saturation preserves accepted work and maps into existing bounded recovery/deferral rather than permanent failure.
- Reuse files and exact replay on restart; missing/corrupt/incompatible files, invalid source identity or missing history require rebuild.
- Index winning replicated state locally and coordinate initial replica data copy/catch-up. Never transfer a source node's local cursor or infer readiness from replication completion.
- Fresh replicas/restores rebuild before serving fulltext; authoritative data/schema are sufficient backups. Source restore invalidates previous index generations even if record IDs and paths match.
- Coordinate shadow build/catch-up, durable selection, activation, rollback, bounded retirement/cleanup retries and reader lifetime across crash.
- Prove real worker handoff, close/quiescence and cross-worker reader refresh, including commit followed by owner death before notification.
- Whole-record TTL/cache eviction removes local search eligibility using the shared local-only action.
- Keep operator-controlled contained native paths, restrictive permissions, disk headroom, encryption prerequisites and safe public errors.
- Expose health, generation, queue/lag, searchable checkpoint, rebuild and cleanup state. Missing/unready indexes fail explicitly, never as an empty success.
- Backend failure must not suppress other consumers or undo a durable source commit; Harper's existing lag/admission policy remains authoritative.
- Rotate the local database incarnation in native generation identity on restore. Precreate secure owned directories and validate canonical path containment, symlinks and platform aliases before open/cleanup.
## Design
[Native Tantivy storage and Harper derived indexes](https://github.com/HarperFast/fulltext/blob/codex/native-storage-design/docs/native-storage-integration.md). Harper owns source state and derived coordination; Tantivy uses native local files. This updates the design, not implementation status.
Comment generated by kAIle (GPT-6)
Contributor guide
Assessment
This issue has not been assessed yet.