apache / apache/arrow-go

[Go] ipc.Writer Option to skip appending data buffers

Open
#76 3 comments 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
Assembly
Stars
404
Forks
145
Avg merge
2d 4h
Merged PRs (30d)
87

Description

For cases where we have a known shared memory region, it would be great if the ipc.Writer (and by extension ipc.Reader?) had the ability to write out everything but the actual buffers holding the data. That way we can still utilize the ipc mechanisms to communicate without having to serialize all the underlying data across the wire.

 

This seems like it should be possible since the `RecordBatch` flatbuffers only contain the metadata and the underlying data buffers are appended later. We just need to skip appending the underlying data buffers.

 

@sbinet thoughts?

**Reporter**: [Nick Poorman](https://issues.apache.org/jira/browse/ARROW-6107) / @nickpoorman

**Note**: *This issue was originally created as [ARROW-6107](https://issues.apache.org/jira/browse/ARROW-6107). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reading the ipc.Writer and ipc.Reader entry points and tracing how RecordBatch flatbuffer metadata is followed by the underlying data buffers. Determine how an option could preserve the IPC metadata path while omitting those buffers for a known shared-memory region; done means the intended writer behavior is defined and covered by the relevant IPC tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.