HarperFast / HarperFast/harper
Add @fullText schema declaration, validation, activation, and introspection
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Outcome
Declare native full-text indexes derived from Harper record fields using the existing schema and derived-index lifecycle conventions.
## Storage and recovery
Harper records in RocksDB are authoritative. Each @fullText declaration materializes an independent Tantivy index in native local files; the index is not stored in RocksDB. Harper transaction logs drive post-commit updates on source nodes and replicas. A node reuses its local files and replays after the durable cursor on restart, or rebuilds locally before serving full-text queries when files are missing, incompatible, or corrupt.
LMDB rejects @fullText activation with an actionable error.
## Acceptance criteria
- @fullText declares weighted fields and English settings without overloading @indexed or @computed; no historical fulltext @indexed type needs compatibility.
- Positions and surface terms remain configurable with approved defaults.
- Synonyms and highlighting remain configurable and disabled by default.
- Source fields, analyzer, weights, compatibility, and feature combinations are validated before activation.
- Multiple declarations per table are independent and can be written in parallel under the shared derived-index runtime.
- Full-text tables explicitly enable audit logging so the transaction log can recover the derived index.
- Paths, storage providers, flush controls, and disk policy are not customer schema fields.
- Native artifact/platform support and local directory access are validated before release qualification.
- Introspection and generated API metadata describe capabilities and readiness; rebuilding indexes do not serve queries.
## Implementation status
- [Add a full-text derived-index backend #2569](https://github.com/HarperFast/harper/pull/2569) implements the Harper adapter and native generation lifecycle.
- [Add @fullText schema declaration and validation #2615](https://github.com/HarperFast/harper/pull/2615) implements schema compilation and validation.
- [Activate native full-text derived indexes #2616](https://github.com/HarperFast/harper/pull/2616) wires schema declarations into the shared HNSW/full-text runtime, native storage, restart recovery, and rebuild flow.
Query integration, introspection, an exact-pinned published Fulltext package with packed-platform validation, and a process-wide native resource governor remain follow-up work. Blob is retained as a valid future source type but activation currently rejects it until bounded asynchronous extraction is available.
## Design
[Native Tantivy storage and Harper derived indexes](https://github.com/HarperFast/fulltext/blob/codex/native-storage-design/docs/native-storage-integration.md)
Comment generated by kAIle (GPT-5)
Contributor guide
Assessment
This issue has not been assessed yet.