apache / apache/arrow

What is the correct way to use IPC if you implement your own IO layer?

Open
#13,978 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

I'm trying to use arrow IPC in C++ with a custom IO layer. I'm wondering what is the correct way to reclaim memory that is allocated by `arrow::io::InputStream::Read(int64_t)`. This returns a `std::shared_ptr`. The arrow documentation is contradictory on whether `std::shared_ptr` owns its own memory.

Here it says that it doesn't own its memory: https://arrow.apache.org/docs/cpp/api/memory.html#_CPPv4N5arrow6BufferE
Here it say it does own its own memory: https://arrow.apache.org/docs/cpp/api/io.html#_CPPv4N5arrow2io8Writable5WriteERKNSt10shared_ptrI6BufferEE

If `arrow::Buffer` doesn't own its own memory, how do you know that arrow has finished using the memory contained in the buffer to free it? This is particularly useful in streaming applications of large amount of arrow ipc. Does arrow ever build references/pointers into the buffer that last beyond the lifetime of the `std::shared_ptr`? It seems that it can if the `std::shared_ptr`'s deconstruction doesn't deallocate the memory underlying the buffer. This seems to be implied to be the case depending on how you read the lifetime requirement on the bytes passed into this consume function: https://arrow.apache.org/docs/cpp/api/ipc.html#_CPPv4N5arrow3ipc13StreamDecoder7ConsumeEPK7uint8_t7int64_t
If you read it as "the bytes passed in must be valid for all record batch processing" as in until the stream of bytes is done. Or you could read it as "these bytes must be valid until the bytes fed pass `next_required_size()` many"

Is there an example of how to do this? If not what is the recommended way of reclaiming memory for processed batches that no longer have references in arrow that was allocated in the IO layer?

Thanks!

Contributor guide

Open the contributing guide

Research direction

Start with the C++ API documentation for arrow::Buffer, arrow::io::InputStream::Read(int64_t), Writable::Write, and ipc::StreamDecoder::Consume. Trace the documented ownership and lifetime requirements, including next_required_size(), and check whether the pages agree. Done means the ownership rules are consistent and the recommended memory-reclamation pattern or example for custom IO and streaming IPC is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.