anitnilay20 / anitnilay20/thoth
Epic: DuckDB-powered native file query engine
- Lingua principale
- Rust
- Stelle
- 70
- Fork
- 5
- Merge medio
- 10h 33m
- PR unite (30g)
- 2
Descrizione
## Goal
Replace the planned custom query/sort/aggregation implementations with DuckDB as the host-side query engine for all locally-opened files. DuckDB is embedded natively in the host binary (no WASM, no server) via the `duckdb` Rust crate.
## Why DuckDB
Issues #53, #54, and #55 each proposed building a custom parser + evaluator for filtering, sorting, and aggregation. DuckDB gives all three for free via standard SQL, plus:
- Native readers for JSON, CSV, Parquet, Arrow, Excel, Avro — no plugin needed for common formats
- Full SQL: `WHERE`, `ORDER BY`, `GROUP BY`, window functions, `HAVING`, `JOIN`
- Lazy scanning — DuckDB does not load the whole file into memory
- `read_json_auto()`, `read_csv_auto()` infer schema automatically
- ~7MB binary footprint
## Architecture
```
Native formats (JSON/CSV/Parquet/…)
File ──► DuckDB (direct scan via read_json_auto / read_csv_auto / …)
Plugin-loaded formats (custom binary, proprietary)
File ──► file-loader plugin (get-range) ──► DuckDB in-memory table
Both paths
DuckDB ──► SQL query ──► dataset-bus ──► data-view render node ──► viewer
```
The viewer has two modes: **raw** (existing lazy scroll via file-loader) and **query** (DuckDB result set). Switching to query mode does not break the existing file-loader flow.
## Sub-issues
- [ ] #148 DuckDB host integration — FileQueryEngine, per-tab connection, format ingestion _(foundation)_
- [ ] #149 SQL query editor UI — query bar, results panel, mode toggle _(needs #148)_
- [ ] #53 Filtering — WHERE clause _(needs #148, #149)_
- [ ] #54 Sorting — ORDER BY + column header click _(needs #148, #149)_
- [ ] #55 Aggregation — GROUP BY + aggregate functions _(needs #148, #149)_
## Dependency order
```
#148 FileQueryEngine ──► #149 Query editor UI ──► #53 Filter
──► #54 Sort
──► #55 Aggregation
```
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.