apache / apache/arrow-nanoarrow

to_arrow_ipc() output missing Arrow IPC end-of-stream (EOS) marker

Open
#867 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
249
Forks
68
Avg merge
2d 21h
Merged PRs (30d)
11

Description

## Summary

The output from nanoarrow's `to_arrow_ipc()` DuckDB extension does not include the Arrow IPC end-of-stream (EOS) marker (`0xFFFFFFFF 0x00000000`). This causes consumers that expect a well-terminated IPC stream to fail.

## Reproduction

When using `to_arrow_ipc()` from the nanoarrow DuckDB extension and concatenating the resulting chunks, the stream lacks the 8-byte EOS sentinel defined by the [Arrow IPC spec](https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format):

> The end of a stream is indicated by a 0 length and 0 metadata length.

For example, feeding the output to DuckDB-WASM's `insertArrowFromIPCStream` fails with:

```
Header-type of flatbuffer-encoded Message is not RecordBatch
```

## Workaround

Manually appending the EOS marker after concatenating chunks resolves the issue:

```js
chunks.push(Buffer.from([0xFF, 0xFF, 0xFF, 0xFF, 0x00, 0x00, 0x00, 0x00]));
return Buffer.concat(chunks);
```

## Expected behavior

`to_arrow_ipc()` should produce a complete, spec-compliant IPC stream that includes the EOS marker, so consumers don't need to append it manually.

## Context

Discussion originated in duckdb/duckdb-node-neo#45. The [Mosaic project](https://github.com/uwdata/mosaic/pull/993) also uses this `to_arrow_ipc()` + nanoarrow pattern.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the nanoarrow DuckDB extension's to_arrow_ipc() entry point and compare its concatenated output with the Arrow IPC streaming specification. Reproduce the failure with DuckDB-WASM's insertArrowFromIPCStream, then verify that the resulting stream includes the 8-byte EOS marker and is accepted without the manual workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.