anitnilay20 / anitnilay20/thoth

DuckDB: host integration and FileQueryEngine

Aperta
#148 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
architecture priority:high size:medium
Lingua principale
Rust
Stelle
70
Fork
5
Merge medio
10h 33m
PR unite (30g)
2

Descrizione

## Context

Core infrastructure for the DuckDB file query engine epic. All other sub-issues (filtering #53, sorting #54, aggregation #55, query editor UI) depend on this.

## What to build

A `FileQueryEngine` struct owned by the host, one instance per open file tab.

### Format detection

On file open, detect whether DuckDB can read the format natively:

| Format | DuckDB function |
|--------|----------------|
| JSON / NDJSON | `read_json_auto('path')` |
| CSV / TSV | `read_csv_auto('path')` |
| Parquet | `read_parquet('path')` |
| Arrow IPC | `read_arrow('path')` |
| Excel | `st_read('path')` (spatial ext) |
| Other | fallback: `get-range` ingestion |

For native formats, register a view: `CREATE VIEW data AS SELECT * FROM read_json_auto('/path/to/file.json')`. DuckDB scans lazily — no full load upfront.

For plugin-loaded formats, bulk-pull via `get-range` and insert into an in-memory table. Show a progress indicator during ingestion; do this async off the UI thread.

### FileQueryEngine API

```rust
pub struct FileQueryEngine { /* DuckDB connection, ingestion state */ }

impl FileQueryEngine {
pub fn open_native(path: &Path, format: NativeFormat) -> Result;
pub fn ingest_from_plugin(records: Vec) -> Result;
pub fn query(&self, sql: &str) -> Result;
pub fn schema(&self) -> Result>; // for editor autocomplete
}
```

### Changes

- Add `duckdb` crate to `Cargo.toml` (feature flags: `bundled` for static linking)
- `src/query/engine.rs` — `FileQueryEngine`
- `src/query/mod.rs` — format detection, ingestion coordinator
- Host file-open path: after plugin `open()`, attempt native DuckDB open; fall back to plugin ingestion

## Acceptance criteria

- [ ] DuckDB opens JSON, CSV, Parquet files directly without plugin ingestion
- [ ] Plugin-loaded formats ingest via `get-range` into an in-memory table async
- [ ] `query(sql)` returns typed rows (not stringified)
- [ ] `schema()` returns column names and DuckDB-inferred types
- [ ] One DuckDB connection per tab; connection dropped when tab closes
- [ ] No UI freeze during ingestion (async worker)

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.