anitnilay20 / anitnilay20/thoth
DuckDB: host integration and FileQueryEngine
- Lingua principale
- Rust
- Stelle
- 70
- Fork
- 5
- Merge medio
- 10h 33m
- PR unite (30g)
- 2
Descrizione
## Context
Core infrastructure for the DuckDB file query engine epic. All other sub-issues (filtering #53, sorting #54, aggregation #55, query editor UI) depend on this.
## What to build
A `FileQueryEngine` struct owned by the host, one instance per open file tab.
### Format detection
On file open, detect whether DuckDB can read the format natively:
| Format | DuckDB function |
|--------|----------------|
| JSON / NDJSON | `read_json_auto('path')` |
| CSV / TSV | `read_csv_auto('path')` |
| Parquet | `read_parquet('path')` |
| Arrow IPC | `read_arrow('path')` |
| Excel | `st_read('path')` (spatial ext) |
| Other | fallback: `get-range` ingestion |
For native formats, register a view: `CREATE VIEW data AS SELECT * FROM read_json_auto('/path/to/file.json')`. DuckDB scans lazily — no full load upfront.
For plugin-loaded formats, bulk-pull via `get-range` and insert into an in-memory table. Show a progress indicator during ingestion; do this async off the UI thread.
### FileQueryEngine API
```rust
pub struct FileQueryEngine { /* DuckDB connection, ingestion state */ }
impl FileQueryEngine {
pub fn open_native(path: &Path, format: NativeFormat) -> Result;
pub fn ingest_from_plugin(records: Vec) -> Result;
pub fn query(&self, sql: &str) -> Result;
pub fn schema(&self) -> Result>; // for editor autocomplete
}
```
### Changes
- Add `duckdb` crate to `Cargo.toml` (feature flags: `bundled` for static linking)
- `src/query/engine.rs` — `FileQueryEngine`
- `src/query/mod.rs` — format detection, ingestion coordinator
- Host file-open path: after plugin `open()`, attempt native DuckDB open; fall back to plugin ingestion
## Acceptance criteria
- [ ] DuckDB opens JSON, CSV, Parquet files directly without plugin ingestion
- [ ] Plugin-loaded formats ingest via `get-range` into an in-memory table async
- [ ] `query(sql)` returns typed rows (not stringified)
- [ ] `schema()` returns column names and DuckDB-inferred types
- [ ] One DuckDB connection per tab; connection dropped when tab closes
- [ ] No UI freeze during ingestion (async worker)
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.