anitnilay20 / anitnilay20/thoth

Epic: DuckDB-powered native file query engine

Open
#147 0 comments 0 reactions 0 assignees View on GitHub
architecture epic feature priority:high size:large
Dominant language
Rust
Stars
70
Forks
5
Avg merge
10h 33m
Merged PRs (30d)
2

Description

## Goal

Replace the planned custom query/sort/aggregation implementations with DuckDB as the host-side query engine for all locally-opened files. DuckDB is embedded natively in the host binary (no WASM, no server) via the `duckdb` Rust crate.

## Why DuckDB

Issues #53, #54, and #55 each proposed building a custom parser + evaluator for filtering, sorting, and aggregation. DuckDB gives all three for free via standard SQL, plus:

- Native readers for JSON, CSV, Parquet, Arrow, Excel, Avro — no plugin needed for common formats
- Full SQL: `WHERE`, `ORDER BY`, `GROUP BY`, window functions, `HAVING`, `JOIN`
- Lazy scanning — DuckDB does not load the whole file into memory
- `read_json_auto()`, `read_csv_auto()` infer schema automatically
- ~7MB binary footprint

## Architecture

```
Native formats (JSON/CSV/Parquet/…)
File ──► DuckDB (direct scan via read_json_auto / read_csv_auto / …)

Plugin-loaded formats (custom binary, proprietary)
File ──► file-loader plugin (get-range) ──► DuckDB in-memory table

Both paths
DuckDB ──► SQL query ──► dataset-bus ──► data-view render node ──► viewer
```

The viewer has two modes: **raw** (existing lazy scroll via file-loader) and **query** (DuckDB result set). Switching to query mode does not break the existing file-loader flow.

## Sub-issues

- [ ] #148 DuckDB host integration — FileQueryEngine, per-tab connection, format ingestion _(foundation)_
- [ ] #149 SQL query editor UI — query bar, results panel, mode toggle _(needs #148)_
- [ ] #53 Filtering — WHERE clause _(needs #148, #149)_
- [ ] #54 Sorting — ORDER BY + column header click _(needs #148, #149)_
- [ ] #55 Aggregation — GROUP BY + aggregate functions _(needs #148, #149)_

## Dependency order

```
#148 FileQueryEngine ──► #149 Query editor UI ──► #53 Filter
──► #54 Sort
──► #55 Aggregation
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.