anitnilay20 / anitnilay20/thoth

DuckDB: host integration and FileQueryEngine

Offen
#148 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
architecture priority:high size:medium
Vorherrschende Sprache
Rust
Sterne
70
Forks
5
Ø Merge
10 Std. 33 Min.
Gemergte PRs (30 T.)
2

Beschreibung

## Context

Core infrastructure for the DuckDB file query engine epic. All other sub-issues (filtering #53, sorting #54, aggregation #55, query editor UI) depend on this.

## What to build

A `FileQueryEngine` struct owned by the host, one instance per open file tab.

### Format detection

On file open, detect whether DuckDB can read the format natively:

| Format | DuckDB function |
|--------|----------------|
| JSON / NDJSON | `read_json_auto('path')` |
| CSV / TSV | `read_csv_auto('path')` |
| Parquet | `read_parquet('path')` |
| Arrow IPC | `read_arrow('path')` |
| Excel | `st_read('path')` (spatial ext) |
| Other | fallback: `get-range` ingestion |

For native formats, register a view: `CREATE VIEW data AS SELECT * FROM read_json_auto('/path/to/file.json')`. DuckDB scans lazily — no full load upfront.

For plugin-loaded formats, bulk-pull via `get-range` and insert into an in-memory table. Show a progress indicator during ingestion; do this async off the UI thread.

### FileQueryEngine API

```rust
pub struct FileQueryEngine { /* DuckDB connection, ingestion state */ }

impl FileQueryEngine {
pub fn open_native(path: &Path, format: NativeFormat) -> Result;
pub fn ingest_from_plugin(records: Vec) -> Result;
pub fn query(&self, sql: &str) -> Result;
pub fn schema(&self) -> Result>; // for editor autocomplete
}
```

### Changes

- Add `duckdb` crate to `Cargo.toml` (feature flags: `bundled` for static linking)
- `src/query/engine.rs` — `FileQueryEngine`
- `src/query/mod.rs` — format detection, ingestion coordinator
- Host file-open path: after plugin `open()`, attempt native DuckDB open; fall back to plugin ingestion

## Acceptance criteria

- [ ] DuckDB opens JSON, CSV, Parquet files directly without plugin ingestion
- [ ] Plugin-loaded formats ingest via `get-range` into an in-memory table async
- [ ] `query(sql)` returns typed rows (not stringified)
- [ ] `schema()` returns column names and DuckDB-inferred types
- [ ] One DuckDB connection per tab; connection dropped when tab closes
- [ ] No UI freeze during ingestion (async worker)

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.