anitnilay20 / anitnilay20/thoth

Epic: DuckDB-powered native file query engine

Abierto
#147 0 comentarios 0 reacciones 0 asignados Ver en GitHub
architecture epic feature priority:high size:large
Lenguaje dominante
Rust
Estrellas
70
Forks
5
Merge medio
10 h 33 min
PR fusionados (30 d)
2

Descripción

## Goal

Replace the planned custom query/sort/aggregation implementations with DuckDB as the host-side query engine for all locally-opened files. DuckDB is embedded natively in the host binary (no WASM, no server) via the `duckdb` Rust crate.

## Why DuckDB

Issues #53, #54, and #55 each proposed building a custom parser + evaluator for filtering, sorting, and aggregation. DuckDB gives all three for free via standard SQL, plus:

- Native readers for JSON, CSV, Parquet, Arrow, Excel, Avro — no plugin needed for common formats
- Full SQL: `WHERE`, `ORDER BY`, `GROUP BY`, window functions, `HAVING`, `JOIN`
- Lazy scanning — DuckDB does not load the whole file into memory
- `read_json_auto()`, `read_csv_auto()` infer schema automatically
- ~7MB binary footprint

## Architecture

```
Native formats (JSON/CSV/Parquet/…)
File ──► DuckDB (direct scan via read_json_auto / read_csv_auto / …)

Plugin-loaded formats (custom binary, proprietary)
File ──► file-loader plugin (get-range) ──► DuckDB in-memory table

Both paths
DuckDB ──► SQL query ──► dataset-bus ──► data-view render node ──► viewer
```

The viewer has two modes: **raw** (existing lazy scroll via file-loader) and **query** (DuckDB result set). Switching to query mode does not break the existing file-loader flow.

## Sub-issues

- [ ] #148 DuckDB host integration — FileQueryEngine, per-tab connection, format ingestion _(foundation)_
- [ ] #149 SQL query editor UI — query bar, results panel, mode toggle _(needs #148)_
- [ ] #53 Filtering — WHERE clause _(needs #148, #149)_
- [ ] #54 Sorting — ORDER BY + column header click _(needs #148, #149)_
- [ ] #55 Aggregation — GROUP BY + aggregate functions _(needs #148, #149)_

## Dependency order

```
#148 FileQueryEngine ──► #149 Query editor UI ──► #53 Filter
──► #54 Sort
──► #55 Aggregation
```

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.