[Go] ipc.Writer Option to skip appending data buffers
- Dominant language
- Assembly
- Stars
- 404
- Forks
- 145
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 87
Description
For cases where we have a known shared memory region, it would be great if the ipc.Writer (and by extension ipc.Reader?) had the ability to write out everything but the actual buffers holding the data. That way we can still utilize the ipc mechanisms to communicate without having to serialize all the underlying data across the wire.
This seems like it should be possible since the `RecordBatch` flatbuffers only contain the metadata and the underlying data buffers are appended later. We just need to skip appending the underlying data buffers.
@sbinet thoughts?
**Reporter**: [Nick Poorman](https://issues.apache.org/jira/browse/ARROW-6107) / @nickpoorman
**Note**: *This issue was originally created as [ARROW-6107](https://issues.apache.org/jira/browse/ARROW-6107). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reading the ipc.Writer and ipc.Reader entry points and tracing how RecordBatch flatbuffer metadata is followed by the underlying data buffers. Determine how an option could preserve the IPC metadata path while omitting those buffers for a known shared-memory region; done means the intended writer behavior is defined and covered by the relevant IPC tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100