apache / apache/arrow

[R Package] map_batches in parallel

Open
#12,919 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

Regular user of disk.frame. I love that packages map function, which allows to apply custom R functions on a larger than ram dataset in paralllel over the batches by using the future package. Am I right map_batches right now supports single core processing as it is implemented as a while loop over the batches? Do you plan to bring this very handy capability to R arrow, too? Sure it must be slower than arrows c++ based functionality, but it is still super useful.

Contributor guide

Open the contributing guide

Research direction

Start at the R package's map_batches entry point and inspect how it iterates over batches, including the stated while-loop behavior and use of the future package. Compare that behavior with the requested parallel capability in R Arrow; done would require an agreed scope for parallel batch processing, not just confirmation of current behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.