apache / apache/arrow-go

[Go] Estimate physical size of data accumulated in array.Builder

Open
#42 0 comments 0 reactions 0 assignees View on GitHub
Type: usage
Dominant language
Assembly
Stars
404
Forks
145
Avg merge
2d 4h
Merged PRs (30d)
87

Description

### Describe the usage question you have. Please include as many useful details as possible.

I'm working on a streaming service that accumulates blocks of Arrow data, dumps them into Arrow Streaming IPC format and then writes them into network socket. The data is accumulated in `[]array.Builder`, which converts into `[]arrow.Array`, then into a single `arrow.Record` and finally into `bytes.Buffer`.

The key part of this algorithm is the separation of a new block: we need to know when the previous block is large enough (but not too large - because it could exceed message size limits in this case) to let us stop data accumulation in `[]array.Builder`, serialize existing data into `bytes.Buffer` and send over the network.

So is it possible to estimate the number of bytes accumulated physically in `array.Builder`? Or may be it even possible to forecast the size of a future `bytes.Buffer` containing serialized data from `[]array.Builder`?

### Component(s)

Go

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.