HarperFast / HarperFast/harper

Implement full-text generation lifecycle, rebuild, eviction, backup, and status

Open
#2,510 1 comment 0 reactions 1 assignee Claimed by @kylebernhardy View on GitHub
area:search area:storage
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Outcome

Connect native Tantivy files to Harper's shared derived runtime and database/index lifecycle.

## Acceptance criteria

- Use the existing elected transaction-log runner, not direct per-worker aftercommit delivery; resolve current committed primary projections.
- Adapt native open/publish/close to FullTextDerivedIndexBackend and store the exact offered cursor inside the Tantivy commit payload.
- Verify native queue saturation preserves accepted work and maps into existing bounded recovery/deferral rather than permanent failure.
- Reuse files and exact replay on restart; missing/corrupt/incompatible files, invalid source identity or missing history require rebuild.
- Index winning replicated state locally and coordinate initial replica data copy/catch-up. Never transfer a source node's local cursor or infer readiness from replication completion.
- Fresh replicas/restores rebuild before serving fulltext; authoritative data/schema are sufficient backups. Source restore invalidates previous index generations even if record IDs and paths match.
- Coordinate shadow build/catch-up, durable selection, activation, rollback, bounded retirement/cleanup retries and reader lifetime across crash.
- Prove real worker handoff, close/quiescence and cross-worker reader refresh, including commit followed by owner death before notification.
- Whole-record TTL/cache eviction removes local search eligibility using the shared local-only action.
- Keep operator-controlled contained native paths, restrictive permissions, disk headroom, encryption prerequisites and safe public errors.
- Expose health, generation, queue/lag, searchable checkpoint, rebuild and cleanup state. Missing/unready indexes fail explicitly, never as an empty success.
- Backend failure must not suppress other consumers or undo a durable source commit; Harper's existing lag/admission policy remains authoritative.

- Rotate the local database incarnation in native generation identity on restore. Precreate secure owned directories and validate canonical path containment, symlinks and platform aliases before open/cleanup.

## Design

[Native Tantivy storage and Harper derived indexes](https://github.com/HarperFast/fulltext/blob/codex/native-storage-design/docs/native-storage-integration.md). Harper owns source state and derived coordination; Tantivy uses native local files. This updates the design, not implementation status.

Comment generated by kAIle (GPT-6)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.