[C++] Utilities to estimate average (de)serialized row size
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
We often parameterize things by number of rows, but what we would rather set the batch size in bytes. This is often the case when reading/writing files or IPC streams. One solution would be to provide utilities to estimate the average row size. For example, the Velox project file readers provide an `estimatedRowSize()` (although I'm not sure how often that is used):
https://github.com/facebookincubator/velox/blob/33c40fda3a7654891c506bf23d078c0da0cd4f0d/velox/dwio/common/Reader.h#L71
This interface for the Parquet reader might be something like:
```cpp
/// \brief Provides average bytes per row as in-memory Arrow data.
///
/// \param sample_rows: if true, will read a sample of rows to estimate the average size of
/// variable length columns
Result EstimateDeserializedRowSize(
parquet::arrow::FileReader file,
std::vector column_indices,
sample_rows = false);
```
Then would be used something like:
```cpp
parquet::arrow::FileReader file = parquet::arrow::OpenFile("path/to/file");
std::vector columns = {1, 3, 5};
int64_t row_size = ARROW_RETURN_NOT_OK(EstimateDeserializedRowSize(file, columns));
// Configure Arrow-specific Parquet reader settings
int64_t batch_size_bytes = 64 * 1024 * 1024;
auto arrow_reader_props = parquet::ArrowReaderProperties();
arrow_reader_props.set_batch_size(batch_size_bytes / row_size);
/// Then use the properties to get a RBR...
```
Similarly, when writing IPC we might want something like:
```cpp
/// \brief Provides average bytes per for in serialized IPC message
Result EstimateSerializedRowSize(
const RecordBatch& batch,
const IpcWriteOptions& write_option,
);
```
So we can use this when writing to a Flight stream:
```cpp
std::shared_ptr table = ...
int64_t row_size = EstimateSerializedRowSize(table->batch(0), write_options);
int64_t batch_size_bytes = 10 * 1024 * 1024;
auto reader = TableBatchReader(table.get());
reader.set_chunksize(batch_size_bytes / row_size);
/// Pass batches to DoPut
```
### Component(s)
C++
Contributor guide
Research direction
Start by reviewing the parquet::arrow::FileReader interface and the RecordBatch/IpcWriteOptions entry points described in the issue. Done means agreeing on and implementing utilities that estimate deserialized Parquet row size and serialized IPC row size for the stated inputs, with tests covering the estimates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100