apache / apache/arrow-dotnet

IPC `ArrowFileWriter` can produce entirely zero-filled encapsulated messages

Open
#409 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
39
Forks
30
Avg merge
1d 9h
Merged PRs (30d)
16

Description

### Describe the bug, including details regarding any error messages, version, and platform.

## Environment

- `Apache.Arrow` / `Apache.Arrow.Compression` 23.0.0, `net472`, x64, Windows 11
- `ArrowFileWriter` over a plain `FileStream`, writing record batches continuously for the duration of a recording

## Background

We are using `Apache.Arrow` to save continuous streaming data from [ONIX hardware](https://open-ephys.github.io/onix-docs/), where our project ([OpenEphys.Onix1](https://github.com/open-ephys/bonsai-onix1)) is hosted as a package that can run in [Bonsai-Rx](https://github.com/bonsai-rx/bonsai).

## Problem

Rarely, on one specific computer, a file comes out with an entire encapsulated message (continuation marker, metadata, and body) replaced by zeros. The footer is intact and its `Block` offsets are correct for every batch, including all batches after the zeroed one, so the file reads until it hits the gap and then raises `ArrowInvalid: Unexpected empty message in IPC file format`. Happens with and without Zstd compression, always early in the recording. Individual valid batches can be read and concatenated together, but this leaves a gap in the recorded samples as confirmed by our clock parameter.

`ArrowFileWriter` takes the block offset from `BaseStream.Position` and the following batch is recorded at the correct offset; we believe this points to Arrow having issued the writes and `FileStream` having accepted them, but the bytes were lost before being saved to disk. We'd like to confirm if this is the case before concluding it's environmental.

We are tracking this in our own issue in case it is how we are utilizing the API. For more details (e.g., byte offsets, and affected files), see open-ephys/bonsai-onix1#683. We have been unable to replicate this on two other systems.

## Our usage

- [`ArrowWriter.cs`](https://github.com/open-ephys/bonsai-onix1/blob/d5b8370/OpenEphys.Onix1/DataFrameWriter/ArrowWriter.cs) — construction, `WriteRecordBatch`, and `WriteEnd` on dispose
- [`ArrowBatchWriter.cs`](https://github.com/open-ephys/bonsai-onix1/blob/d5b8370/OpenEphys.Onix1/DataFrameWriter/ArrowBatchWriter.cs) — batches are accumulated with an Rx `Buffer(timeout, count)`, so `WriteRecordBatch` calls are serialized but arrive on different threads (producer thread on a count flush, thread-pool timer thread on a timeout flush)

Note also that our `ArrowBuffer`s wrap unmanaged memory via a custom `MemoryManager` to handle `OpenCV.Net.Mat` objects.

I'd be happy to answer any questions you might have about our usage, and if there is any other information I can provide to narrow this down please let me know.

## Questions

1. Is there any path in `ArrowFileWriter` that advances `BaseStream.Position` without writing bytes (a seek, or padding emitted by seeking)?
2. Is there anything we should be doing differently? `FileOptions.WriteThrough`, periodic `Flush(true)` on the base stream, or otherwise to keep a lower layer from silently dropping a write like this?

Contributor guide

Open the contributing guide

Research direction

Start by reading ArrowFileWriter and the usage points in ArrowWriter.cs and ArrowBatchWriter.cs, especially BaseStream.Position and the serialized WriteRecordBatch calls from different threads. Reproduce the zero-filled IPC message if possible, comparing compressed and uncompressed writes and the affected offsets. Done means identifying whether ArrowFileWriter can advance the position without writing, or documenting the required stream usage or environmental cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.