anitnilay20 / anitnilay20/thoth

Epic: DuckDB-powered native file query engine

オープン
#147 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
architecture epic feature priority:high size:large
主要言語
Rust
スター
70
フォーク
5
平均マージ
10時間 33分
マージ済み PR(30日)
2

説明

## Goal

Replace the planned custom query/sort/aggregation implementations with DuckDB as the host-side query engine for all locally-opened files. DuckDB is embedded natively in the host binary (no WASM, no server) via the `duckdb` Rust crate.

## Why DuckDB

Issues #53, #54, and #55 each proposed building a custom parser + evaluator for filtering, sorting, and aggregation. DuckDB gives all three for free via standard SQL, plus:

- Native readers for JSON, CSV, Parquet, Arrow, Excel, Avro — no plugin needed for common formats
- Full SQL: `WHERE`, `ORDER BY`, `GROUP BY`, window functions, `HAVING`, `JOIN`
- Lazy scanning — DuckDB does not load the whole file into memory
- `read_json_auto()`, `read_csv_auto()` infer schema automatically
- ~7MB binary footprint

## Architecture

```
Native formats (JSON/CSV/Parquet/…)
File ──► DuckDB (direct scan via read_json_auto / read_csv_auto / …)

Plugin-loaded formats (custom binary, proprietary)
File ──► file-loader plugin (get-range) ──► DuckDB in-memory table

Both paths
DuckDB ──► SQL query ──► dataset-bus ──► data-view render node ──► viewer
```

The viewer has two modes: **raw** (existing lazy scroll via file-loader) and **query** (DuckDB result set). Switching to query mode does not break the existing file-loader flow.

## Sub-issues

- [ ] #148 DuckDB host integration — FileQueryEngine, per-tab connection, format ingestion _(foundation)_
- [ ] #149 SQL query editor UI — query bar, results panel, mode toggle _(needs #148)_
- [ ] #53 Filtering — WHERE clause _(needs #148, #149)_
- [ ] #54 Sorting — ORDER BY + column header click _(needs #148, #149)_
- [ ] #55 Aggregation — GROUP BY + aggregate functions _(needs #148, #149)_

## Dependency order

```
#148 FileQueryEngine ──► #149 Query editor UI ──► #53 Filter
──► #54 Sort
──► #55 Aggregation
```

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。