apache / apache/arrow-nanoarrow
to_arrow_ipc() output missing Arrow IPC end-of-stream (EOS) marker
- Dominant language
- C
- Stars
- 249
- Forks
- 68
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 11
Description
## Summary
The output from nanoarrow's `to_arrow_ipc()` DuckDB extension does not include the Arrow IPC end-of-stream (EOS) marker (`0xFFFFFFFF 0x00000000`). This causes consumers that expect a well-terminated IPC stream to fail.
## Reproduction
When using `to_arrow_ipc()` from the nanoarrow DuckDB extension and concatenating the resulting chunks, the stream lacks the 8-byte EOS sentinel defined by the [Arrow IPC spec](https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format):
> The end of a stream is indicated by a 0 length and 0 metadata length.
For example, feeding the output to DuckDB-WASM's `insertArrowFromIPCStream` fails with:
```
Header-type of flatbuffer-encoded Message is not RecordBatch
```
## Workaround
Manually appending the EOS marker after concatenating chunks resolves the issue:
```js
chunks.push(Buffer.from([0xFF, 0xFF, 0xFF, 0xFF, 0x00, 0x00, 0x00, 0x00]));
return Buffer.concat(chunks);
```
## Expected behavior
`to_arrow_ipc()` should produce a complete, spec-compliant IPC stream that includes the EOS marker, so consumers don't need to append it manually.
## Context
Discussion originated in duckdb/duckdb-node-neo#45. The [Mosaic project](https://github.com/uwdata/mosaic/pull/993) also uses this `to_arrow_ipc()` + nanoarrow pattern.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the nanoarrow DuckDB extension's to_arrow_ipc() entry point and compare its concatenated output with the Arrow IPC streaming specification. Reproduce the failure with DuckDB-WASM's insertArrowFromIPCStream, then verify that the resulting stream includes the 8-byte EOS marker and is accepted without the manual workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100