apache / apache/arrow

[C++] Create DataHolder that can be used for caching during exec plans

Open
#29,901 0 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: task
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

The purpose of this task is to make an ExecNode that can provide the following functionality.
1. Be able to obtain heuristics about our memory consumption and have a memory consumption threshold

1. Be able to write incoming ExecBatch to disk if memory consumption is above the threshold, stores either the ExecBatch or a handle to file in a queue.

1. Provide an api for pulling an ExecBatch from the queue. It should favor pulling all of the batches that are in memory first and then the ones that are handles to files.

 

PRs to reference



 

**Reporter**: [Alexander Ocsa](https://issues.apache.org/jira/browse/ARROW-14330) / @aocsa
#### PRs and other links:
- [GitHub Pull Request #11426](https://github.com/apache/arrow/pull/11426)

**Note**: *This issue was originally created as [ARROW-14330](https://issues.apache.org/jira/browse/ARROW-14330). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the referenced PR #11017 and PR #11426, then trace the ExecNode and ExecBatch concepts involved. Done means the design provides memory heuristics and a threshold, spills incoming batches to disk when needed, queues batches or file handles, and retrieves in-memory batches before spilled ones.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.