dfe-analytical-services / dfe-analytical-services/analysts-guide

Recommend that analysts using databricks load the arrow package for more efficient sparklyr / DBI processes

Open
#141 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
R
Stars
6
Forks
9
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
This is weird, but if you do `library(arrow)` to attach the arrow package then a bunch of other stuff from sparklyr and DBI is impressively faster.

**Describe the solution you'd like**
A clearly highlighted note or callout box highlighting this as something all analysts should generally do by default as it gives some stonking speed improvements for almost no cost.

**Describe alternatives you've considered**
None - we should make it clear in the guidance for analysts to know about.

**Additional context**
More details and some benchmarks on https://arrow.apache.org/blog/2019/01/25/r-spark-improvements/, though I can also personally vouch for this, having a notebook in a workflow that took around 30mins dropping now to just a couple of minutes!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.