HarperFast / HarperFast/fulltext

Support concurrent native indexes with bounded shared resources

Open
#13 0 comments 0 reactions 1 assignee Claimed by @kylebernhardy View on GitHub
area:native area:performance
Dominant language
Rust
Stars
0
Forks
0
Avg merge
3h 53m
Merged PRs (30d)
18

Description

## Outcome

Allow multiple native Tantivy indexes to ingest and search concurrently while protecting the hosting process.

## Acceptance criteria

- One writer per canonical physical directory/generation; independent index writers may progress in parallel.
- Bound process-wide resident writers, indexing/merge threads, writer memory, queued bytes, searches, mappings and rebuild generations.
- Measure engine-owned threads and memory; per-index limits or bounded command queues alone do not establish aggregate limits.
- Preserve fair query, recovery and shutdown progress under sustained ingestion and merge load.
- Qualify worker attachments and path aliases, owner loss, close/reopen, and generation retirement with no live-handle reuse.
- Measure actual physical/virtual memory and shared file pages rather than assuming each Node worker duplicates index contents.
- Measure achieved ingestion and lag/backlog drain together; account for merge/rebuild disk headroom.
- Harper controls source admission and disk policy. No host-storage wait pool, Rocks transaction lane or wrapper lease service is introduced.

## Design

[Native Tantivy storage and Harper derived indexes](https://github.com/HarperFast/fulltext/blob/codex/native-storage-design/docs/native-storage-integration.md). Native Tantivy files are the only delivery target for standalone use and Harper. Proposed APIs and integration gates are not claims of completed implementation.

Comment generated by kAIle (GPT-6)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.