anitnilay20 / anitnilay20/thoth

[Future] Tantivy: embedded full-text search for local files

オープン
#151 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
feature future priority:low size:large
主要言語
Rust
スター
70
フォーク
5
平均マージ
10時間 33分
マージ済み PR(30日)
2

説明

## Idea

Embed [Tantivy](https://github.com/quickwit-oss/tantivy) — a pure-Rust full-text search engine — as a local search index over opened files. This would complement DuckDB (#147 epic): DuckDB handles structured queries (`price > 100`, `GROUP BY status`), Tantivy handles unstructured text search (`'connection refused' AND host:prod`).

Seshat already provides this capability against remote Elasticsearch. Tantivy would bring equivalent power to local files with no external server.

## What it enables

- BM25-ranked full-text search over large log files or document collections
- Fuzzy matching, phrase queries, field-boosting
- Instant results on indexed files (index built once per file, persisted)
- Useful when DuckDB `WHERE message LIKE '%error%'` is too slow or too literal

## Relationship to DuckDB

These are complementary, not competing. A query bar could route automatically:
- Structured expressions → DuckDB SQL
- Free-text phrases → Tantivy

## Explore before committing

- Index persistence strategy (where to store per-file indices)
- Re-index on file change
- Memory footprint for large files
- Whether DuckDB FTS extension covers enough of the use case to make Tantivy unnecessary

## Not prioritised yet

Parking here for future consideration. Revisit once the DuckDB query engine is shipped and actual user feedback shows demand for full-text search beyond what DuckDB's `LIKE` / FTS extension can provide.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。