anitnilay20 / anitnilay20/thoth

[Future] Tantivy: embedded full-text search for local files

Offen
#151 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
feature future priority:low size:large
Vorherrschende Sprache
Rust
Sterne
70
Forks
5
Ø Merge
10 Std. 33 Min.
Gemergte PRs (30 T.)
2

Beschreibung

## Idea

Embed [Tantivy](https://github.com/quickwit-oss/tantivy) — a pure-Rust full-text search engine — as a local search index over opened files. This would complement DuckDB (#147 epic): DuckDB handles structured queries (`price > 100`, `GROUP BY status`), Tantivy handles unstructured text search (`'connection refused' AND host:prod`).

Seshat already provides this capability against remote Elasticsearch. Tantivy would bring equivalent power to local files with no external server.

## What it enables

- BM25-ranked full-text search over large log files or document collections
- Fuzzy matching, phrase queries, field-boosting
- Instant results on indexed files (index built once per file, persisted)
- Useful when DuckDB `WHERE message LIKE '%error%'` is too slow or too literal

## Relationship to DuckDB

These are complementary, not competing. A query bar could route automatically:
- Structured expressions → DuckDB SQL
- Free-text phrases → Tantivy

## Explore before committing

- Index persistence strategy (where to store per-file indices)
- Re-index on file change
- Memory footprint for large files
- Whether DuckDB FTS extension covers enough of the use case to make Tantivy unnecessary

## Not prioritised yet

Parking here for future consideration. Revisit once the DuckDB query engine is shipped and actual user feedback shows demand for full-text search beyond what DuckDB's `LIKE` / FTS extension can provide.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.