anitnilay20 / anitnilay20/thoth

[Future] Arrow IPC: typed columnar format for the plugin dataset bus

Offen
#152 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
architecture future priority:low size:medium
Vorherrschende Sprache
Rust
Sterne
70
Forks
5
Ø Merge
10 Std. 33 Min.
Gemergte PRs (30 T.)
2

Beschreibung

## Idea

Replace the current `list>` wire format in `dataset-bus.publish` with Arrow IPC — a flat typed columnar buffer that crosses the WASM boundary in a single copy instead of thousands of individual string copies.

## Why this matters

The plugin→host databus is the one place in the stack where data crosses a WASM boundary. Every cell in the current format is an individual string allocation and copy through the component model ABI. For plugins that push large datasets (100k+ rows), this becomes measurable overhead.

Arrow IPC replaces the WIT signature from:

```wit
publish: func(name: string, columns: list, rows: list>) -> string;
```

to something like:

```wit
publish-arrow: func(name: string, ipc-bytes: list) -> string;
```

The plugin serialises its data as an Arrow IPC stream; the host deserialises into a `RecordBatch`. DuckDB (already in the host) can scan Arrow record batches natively via `query_arrow()` — no second conversion needed.

## Not urgent

The current string format is fine for typical plugin datasets. The WIT file already notes this as a future direction. Revisit when profiling shows the databus crossing is a real bottleneck — likely only relevant for plugins that stream large, high-frequency datasets (monitoring, Kafka consumers, etc.).

## Prerequisites

- DuckDB native integration (#148) should be shipped first — Arrow's value compounds when DuckDB can consume the batch directly
- Requires Arrow encoding support in the plugin SDK (adds a dependency for plugin authors)
- WIT is a breaking change — needs a minor version bump and migration path for existing plugins

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.