HarperFast / HarperFast/fulltext

Implement bounded native execution and one-writer-per-index coordination

Open
#17 2 comments 0 reactions 1 assignee Claimed by @kylebernhardy View on GitHub
area:native area:performance
Dominant language
Rust
Stars
0
Forks
0
Avg merge
3h 53m
Merged PRs (30d)
18

Description

## Outcome

Run native indexing, merging, publication, and search with bounded admission and deterministic ownership.

## Acceptance criteria

- Exactly one Tantivy IndexWriter per canonical native directory/index/generation; independent indexes can progress in parallel.
- Use bounded native writer/commit lanes and search execution outside the JS event loop and shared libuv pool.
- Enforce aggregate memory, resident writer, native thread, merge, queue and search budgets; current per-index settings are not proof of process-wide bounds.
- Count retained native state during worker loss and preserve control/query completion under saturation.
- Distinguish queued cancellation from started non-cancelable native operations; panic/failure poisons the affected generation.
- Keep Harper runner election and epoch fencing in Harper; native directory locks remain the physical writer guard.
- No synchronous host-storage handoff or blocked-storage-thread pool is required for native MmapDirectory.

## Design

[Native Tantivy storage and Harper derived indexes](https://github.com/HarperFast/fulltext/blob/codex/native-storage-design/docs/native-storage-integration.md). Native Tantivy files are the only delivery target for standalone use and Harper. Proposed APIs and integration gates are not claims of completed implementation.

Comment generated by kAIle (GPT-6)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.