anitnilay20 / anitnilay20/thoth

[Future] Tantivy: embedded full-text search for local files

Abierto
#151 0 comentarios 0 reacciones 0 asignados Ver en GitHub
feature future priority:low size:large
Lenguaje dominante
Rust
Estrellas
70
Forks
5
Merge medio
10 h 33 min
PR fusionados (30 d)
2

Descripción

## Idea

Embed [Tantivy](https://github.com/quickwit-oss/tantivy) — a pure-Rust full-text search engine — as a local search index over opened files. This would complement DuckDB (#147 epic): DuckDB handles structured queries (`price > 100`, `GROUP BY status`), Tantivy handles unstructured text search (`'connection refused' AND host:prod`).

Seshat already provides this capability against remote Elasticsearch. Tantivy would bring equivalent power to local files with no external server.

## What it enables

- BM25-ranked full-text search over large log files or document collections
- Fuzzy matching, phrase queries, field-boosting
- Instant results on indexed files (index built once per file, persisted)
- Useful when DuckDB `WHERE message LIKE '%error%'` is too slow or too literal

## Relationship to DuckDB

These are complementary, not competing. A query bar could route automatically:
- Structured expressions → DuckDB SQL
- Free-text phrases → Tantivy

## Explore before committing

- Index persistence strategy (where to store per-file indices)
- Re-index on file change
- Memory footprint for large files
- Whether DuckDB FTS extension covers enough of the use case to make Tantivy unnecessary

## Not prioritised yet

Parking here for future consideration. Revisit once the DuckDB query engine is shipped and actual user feedback shows demand for full-text search beyond what DuckDB's `LIKE` / FTS extension can provide.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.