IPC `ArrowFileWriter` can produce entirely zero-filled encapsulated messages
- Dominant language
- C#
- Stars
- 39
- Forks
- 30
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 16
Description
### Describe the bug, including details regarding any error messages, version, and platform.
## Environment
- `Apache.Arrow` / `Apache.Arrow.Compression` 23.0.0, `net472`, x64, Windows 11
- `ArrowFileWriter` over a plain `FileStream`, writing record batches continuously for the duration of a recording
## Background
We are using `Apache.Arrow` to save continuous streaming data from [ONIX hardware](https://open-ephys.github.io/onix-docs/), where our project ([OpenEphys.Onix1](https://github.com/open-ephys/bonsai-onix1)) is hosted as a package that can run in [Bonsai-Rx](https://github.com/bonsai-rx/bonsai).
## Problem
Rarely, on one specific computer, a file comes out with an entire encapsulated message (continuation marker, metadata, and body) replaced by zeros. The footer is intact and its `Block` offsets are correct for every batch, including all batches after the zeroed one, so the file reads until it hits the gap and then raises `ArrowInvalid: Unexpected empty message in IPC file format`. Happens with and without Zstd compression, always early in the recording. Individual valid batches can be read and concatenated together, but this leaves a gap in the recorded samples as confirmed by our clock parameter.
`ArrowFileWriter` takes the block offset from `BaseStream.Position` and the following batch is recorded at the correct offset; we believe this points to Arrow having issued the writes and `FileStream` having accepted them, but the bytes were lost before being saved to disk. We'd like to confirm if this is the case before concluding it's environmental.
We are tracking this in our own issue in case it is how we are utilizing the API. For more details (e.g., byte offsets, and affected files), see open-ephys/bonsai-onix1#683. We have been unable to replicate this on two other systems.
## Our usage
- [`ArrowWriter.cs`](https://github.com/open-ephys/bonsai-onix1/blob/d5b8370/OpenEphys.Onix1/DataFrameWriter/ArrowWriter.cs) — construction, `WriteRecordBatch`, and `WriteEnd` on dispose
- [`ArrowBatchWriter.cs`](https://github.com/open-ephys/bonsai-onix1/blob/d5b8370/OpenEphys.Onix1/DataFrameWriter/ArrowBatchWriter.cs) — batches are accumulated with an Rx `Buffer(timeout, count)`, so `WriteRecordBatch` calls are serialized but arrive on different threads (producer thread on a count flush, thread-pool timer thread on a timeout flush)
Note also that our `ArrowBuffer`s wrap unmanaged memory via a custom `MemoryManager` to handle `OpenCV.Net.Mat` objects.
I'd be happy to answer any questions you might have about our usage, and if there is any other information I can provide to narrow this down please let me know.
## Questions
1. Is there any path in `ArrowFileWriter` that advances `BaseStream.Position` without writing bytes (a seek, or padding emitted by seeking)?
2. Is there anything we should be doing differently? `FileOptions.WriteThrough`, periodic `Flush(true)` on the base stream, or otherwise to keep a lower layer from silently dropping a write like this?
Contributor guide
Research direction
Start by reading ArrowFileWriter and the usage points in ArrowWriter.cs and ArrowBatchWriter.cs, especially BaseStream.Position and the serialized WriteRecordBatch calls from different threads. Reproduce the zero-filled IPC message if possible, comparing compressed and uncompressed writes and the affected offsets. Done means identifying whether ArrowFileWriter can advance the position without writing, or documenting the required stream usage or environmental cause.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100