anitnilay20 / anitnilay20/thoth

[Future] Arrow IPC: typed columnar format for the plugin dataset bus

Aperta
#152 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
architecture future priority:low size:medium
Lingua principale
Rust
Stelle
70
Fork
5
Merge medio
10h 33m
PR unite (30g)
2

Descrizione

## Idea

Replace the current `list>` wire format in `dataset-bus.publish` with Arrow IPC — a flat typed columnar buffer that crosses the WASM boundary in a single copy instead of thousands of individual string copies.

## Why this matters

The plugin→host databus is the one place in the stack where data crosses a WASM boundary. Every cell in the current format is an individual string allocation and copy through the component model ABI. For plugins that push large datasets (100k+ rows), this becomes measurable overhead.

Arrow IPC replaces the WIT signature from:

```wit
publish: func(name: string, columns: list, rows: list>) -> string;
```

to something like:

```wit
publish-arrow: func(name: string, ipc-bytes: list) -> string;
```

The plugin serialises its data as an Arrow IPC stream; the host deserialises into a `RecordBatch`. DuckDB (already in the host) can scan Arrow record batches natively via `query_arrow()` — no second conversion needed.

## Not urgent

The current string format is fine for typical plugin datasets. The WIT file already notes this as a future direction. Revisit when profiling shows the databus crossing is a real bottleneck — likely only relevant for plugins that stream large, high-frequency datasets (monitoring, Kafka consumers, etc.).

## Prerequisites

- DuckDB native integration (#148) should be shipped first — Arrow's value compounds when DuckDB can consume the batch directly
- Requires Arrow encoding support in the plugin SDK (adds a dependency for plugin authors)
- WIT is a breaking change — needs a minor version bump and migration path for existing plugins

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.