anitnilay20 / anitnilay20/thoth

Epic: DuckDB-powered native file query engine

Offen
#147 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
architecture epic feature priority:high size:large
Vorherrschende Sprache
Rust
Sterne
70
Forks
5
Ø Merge
10 Std. 33 Min.
Gemergte PRs (30 T.)
2

Beschreibung

## Goal

Replace the planned custom query/sort/aggregation implementations with DuckDB as the host-side query engine for all locally-opened files. DuckDB is embedded natively in the host binary (no WASM, no server) via the `duckdb` Rust crate.

## Why DuckDB

Issues #53, #54, and #55 each proposed building a custom parser + evaluator for filtering, sorting, and aggregation. DuckDB gives all three for free via standard SQL, plus:

- Native readers for JSON, CSV, Parquet, Arrow, Excel, Avro — no plugin needed for common formats
- Full SQL: `WHERE`, `ORDER BY`, `GROUP BY`, window functions, `HAVING`, `JOIN`
- Lazy scanning — DuckDB does not load the whole file into memory
- `read_json_auto()`, `read_csv_auto()` infer schema automatically
- ~7MB binary footprint

## Architecture

```
Native formats (JSON/CSV/Parquet/…)
File ──► DuckDB (direct scan via read_json_auto / read_csv_auto / …)

Plugin-loaded formats (custom binary, proprietary)
File ──► file-loader plugin (get-range) ──► DuckDB in-memory table

Both paths
DuckDB ──► SQL query ──► dataset-bus ──► data-view render node ──► viewer
```

The viewer has two modes: **raw** (existing lazy scroll via file-loader) and **query** (DuckDB result set). Switching to query mode does not break the existing file-loader flow.

## Sub-issues

- [ ] #148 DuckDB host integration — FileQueryEngine, per-tab connection, format ingestion _(foundation)_
- [ ] #149 SQL query editor UI — query bar, results panel, mode toggle _(needs #148)_
- [ ] #53 Filtering — WHERE clause _(needs #148, #149)_
- [ ] #54 Sorting — ORDER BY + column header click _(needs #148, #149)_
- [ ] #55 Aggregation — GROUP BY + aggregate functions _(needs #148, #149)_

## Dependency order

```
#148 FileQueryEngine ──► #149 Query editor UI ──► #53 Filter
──► #54 Sort
──► #55 Aggregation
```

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.