apache / apache/arrow

[C++] Utilities to estimate average (de)serialized row size

Open
#34,712 6 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

We often parameterize things by number of rows, but what we would rather set the batch size in bytes. This is often the case when reading/writing files or IPC streams. One solution would be to provide utilities to estimate the average row size. For example, the Velox project file readers provide an `estimatedRowSize()` (although I'm not sure how often that is used):

https://github.com/facebookincubator/velox/blob/33c40fda3a7654891c506bf23d078c0da0cd4f0d/velox/dwio/common/Reader.h#L71

This interface for the Parquet reader might be something like:

```cpp
/// \brief Provides average bytes per row as in-memory Arrow data.
///
/// \param sample_rows: if true, will read a sample of rows to estimate the average size of
/// variable length columns
Result EstimateDeserializedRowSize(
parquet::arrow::FileReader file,
std::vector column_indices,
sample_rows = false);
```

Then would be used something like:

```cpp
parquet::arrow::FileReader file = parquet::arrow::OpenFile("path/to/file");
std::vector columns = {1, 3, 5};
int64_t row_size = ARROW_RETURN_NOT_OK(EstimateDeserializedRowSize(file, columns));

// Configure Arrow-specific Parquet reader settings
int64_t batch_size_bytes = 64 * 1024 * 1024;
auto arrow_reader_props = parquet::ArrowReaderProperties();
arrow_reader_props.set_batch_size(batch_size_bytes / row_size);

/// Then use the properties to get a RBR...
```

Similarly, when writing IPC we might want something like:

```cpp
/// \brief Provides average bytes per for in serialized IPC message
Result EstimateSerializedRowSize(
const RecordBatch& batch,
const IpcWriteOptions& write_option,
);
```

So we can use this when writing to a Flight stream:

```cpp
std::shared_ptr table = ...
int64_t row_size = EstimateSerializedRowSize(table->batch(0), write_options);

int64_t batch_size_bytes = 10 * 1024 * 1024;
auto reader = TableBatchReader(table.get());
reader.set_chunksize(batch_size_bytes / row_size);

/// Pass batches to DoPut
```

### Component(s)

C++

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the parquet::arrow::FileReader interface and the RecordBatch/IpcWriteOptions entry points described in the issue. Done means agreeing on and implementing utilities that estimate deserialized Parquet row size and serialized IPC row size for the stated inputs, with tests covering the estimates.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.