anitnilay20 / anitnilay20/thoth

DuckDB: host integration and FileQueryEngine

Aberta
#148 0 comentários 0 reações 0 responsáveis Ver no GitHub
architecture priority:high size:medium
Linguagem predominante
Rust
Estrelas
70
Forks
5
Merge médio
10h 33min
PRs com merge (30d)
2

Descrição

## Context

Core infrastructure for the DuckDB file query engine epic. All other sub-issues (filtering #53, sorting #54, aggregation #55, query editor UI) depend on this.

## What to build

A `FileQueryEngine` struct owned by the host, one instance per open file tab.

### Format detection

On file open, detect whether DuckDB can read the format natively:

| Format | DuckDB function |
|--------|----------------|
| JSON / NDJSON | `read_json_auto('path')` |
| CSV / TSV | `read_csv_auto('path')` |
| Parquet | `read_parquet('path')` |
| Arrow IPC | `read_arrow('path')` |
| Excel | `st_read('path')` (spatial ext) |
| Other | fallback: `get-range` ingestion |

For native formats, register a view: `CREATE VIEW data AS SELECT * FROM read_json_auto('/path/to/file.json')`. DuckDB scans lazily — no full load upfront.

For plugin-loaded formats, bulk-pull via `get-range` and insert into an in-memory table. Show a progress indicator during ingestion; do this async off the UI thread.

### FileQueryEngine API

```rust
pub struct FileQueryEngine { /* DuckDB connection, ingestion state */ }

impl FileQueryEngine {
pub fn open_native(path: &Path, format: NativeFormat) -> Result;
pub fn ingest_from_plugin(records: Vec) -> Result;
pub fn query(&self, sql: &str) -> Result;
pub fn schema(&self) -> Result>; // for editor autocomplete
}
```

### Changes

- Add `duckdb` crate to `Cargo.toml` (feature flags: `bundled` for static linking)
- `src/query/engine.rs` — `FileQueryEngine`
- `src/query/mod.rs` — format detection, ingestion coordinator
- Host file-open path: after plugin `open()`, attempt native DuckDB open; fall back to plugin ingestion

## Acceptance criteria

- [ ] DuckDB opens JSON, CSV, Parquet files directly without plugin ingestion
- [ ] Plugin-loaded formats ingest via `get-range` into an in-memory table async
- [ ] `query(sql)` returns typed rows (not stringified)
- [ ] `schema()` returns column names and DuckDB-inferred types
- [ ] One DuckDB connection per tab; connection dropped when tab closes
- [ ] No UI freeze during ingestion (async worker)

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.