apache / apache/arrow

[C++] Implement arrow::Table::GetSizeInBytes() that returns the sum of all Buffer sizes from the internal ArrayData objects

Open
#28,618 11 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

This function will return the sum of `Buffer::size()` as int64 for all the ArrayData objects spread across the Table columns

**Reporter**: [Percy Camilo Triveño Aucahuasi](https://issues.apache.org/jira/browse/ARROW-12888) / @aucahuasi
#### Subtasks:
- [X] [[C++] Create naive implementation of algorithm to estimate table/batch buffer size](https://github.com/apache/arrow/issues/29924)
- [X] [[C++] Create kernel to determine buffer memory "referenced" by arrays (even if there are offsets)](https://github.com/apache/arrow/issues/29925)
- [ ] [[C++] Improve array size estimation to account for shared buffers](https://github.com/apache/arrow/issues/29926)

**Note**: *This issue was originally created as [ARROW-12888](https://issues.apache.org/jira/browse/ARROW-12888). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

No source file or test is named. Start by reviewing the existing Table, ArrayData, and Buffer APIs and the completed subtasks linked in this issue; done means returning an int64 sum of Buffer::size() values across the table while accounting for shared buffers.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.