dfe-analytical-services / dfe-analytical-services/analysts-guide
Recommend that analysts using databricks load the arrow package for more efficient sparklyr / DBI processes
- Dominant language
- R
- Stars
- 6
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
This is weird, but if you do `library(arrow)` to attach the arrow package then a bunch of other stuff from sparklyr and DBI is impressively faster.
**Describe the solution you'd like**
A clearly highlighted note or callout box highlighting this as something all analysts should generally do by default as it gives some stonking speed improvements for almost no cost.
**Describe alternatives you've considered**
None - we should make it clear in the guidance for analysts to know about.
**Additional context**
More details and some benchmarks on https://arrow.apache.org/blog/2019/01/25/r-spark-improvements/, though I can also personally vouch for this, having a notebook in a workflow that took around 30mins dropping now to just a couple of minutes!
Contributor guide
Assessment
This issue has not been assessed yet.