[FEATURE REQUEST]: Easy way to convert Spark DataFrame to Analysis.DataFrame using Arrow
Open
enhancement
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 332
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 9
Description
Hi,
I'm looking for an easy way to convert Spark DF to Analysis DF using Arrow.
I have some cases in which I need to "materialize" Spark DF as fast as possible, do some local transformations, then distribute the payload again.
Based on description this can work pretty nice with Arrow batches between so the overhead can be minimized.
ideally a utility function would look like this:
var fxDataFrame = sparkDataFrame.ToFxDF();
and maybe reversed function as well.
var sparkDataFrame= fxDataFrame .ToSparkDF();
Any thoughts ?
Thanks,
Aleks
Contributor guide
Assessment
This issue has not been assessed yet.