anitnilay20 / anitnilay20/thoth

DuckDB: host integration and FileQueryEngine

Abierto
#148 0 comentarios 0 reacciones 0 asignados Ver en GitHub
architecture priority:high size:medium
Lenguaje dominante
Rust
Estrellas
70
Forks
5
Merge medio
10 h 33 min
PR fusionados (30 d)
2

Descripción

## Context

Core infrastructure for the DuckDB file query engine epic. All other sub-issues (filtering #53, sorting #54, aggregation #55, query editor UI) depend on this.

## What to build

A `FileQueryEngine` struct owned by the host, one instance per open file tab.

### Format detection

On file open, detect whether DuckDB can read the format natively:

| Format | DuckDB function |
|--------|----------------|
| JSON / NDJSON | `read_json_auto('path')` |
| CSV / TSV | `read_csv_auto('path')` |
| Parquet | `read_parquet('path')` |
| Arrow IPC | `read_arrow('path')` |
| Excel | `st_read('path')` (spatial ext) |
| Other | fallback: `get-range` ingestion |

For native formats, register a view: `CREATE VIEW data AS SELECT * FROM read_json_auto('/path/to/file.json')`. DuckDB scans lazily — no full load upfront.

For plugin-loaded formats, bulk-pull via `get-range` and insert into an in-memory table. Show a progress indicator during ingestion; do this async off the UI thread.

### FileQueryEngine API

```rust
pub struct FileQueryEngine { /* DuckDB connection, ingestion state */ }

impl FileQueryEngine {
pub fn open_native(path: &Path, format: NativeFormat) -> Result;
pub fn ingest_from_plugin(records: Vec) -> Result;
pub fn query(&self, sql: &str) -> Result;
pub fn schema(&self) -> Result>; // for editor autocomplete
}
```

### Changes

- Add `duckdb` crate to `Cargo.toml` (feature flags: `bundled` for static linking)
- `src/query/engine.rs` — `FileQueryEngine`
- `src/query/mod.rs` — format detection, ingestion coordinator
- Host file-open path: after plugin `open()`, attempt native DuckDB open; fall back to plugin ingestion

## Acceptance criteria

- [ ] DuckDB opens JSON, CSV, Parquet files directly without plugin ingestion
- [ ] Plugin-loaded formats ingest via `get-range` into an in-memory table async
- [ ] `query(sql)` returns typed rows (not stringified)
- [ ] `schema()` returns column names and DuckDB-inferred types
- [ ] One DuckDB connection per tab; connection dropped when tab closes
- [ ] No UI freeze during ingestion (async worker)

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.