dotnet / dotnet/spark

[FEATURE REQUEST]: Easy way to convert Spark DataFrame to Analysis.DataFrame using Arrow

Open
#962 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

Hi,

I'm looking for an easy way to convert Spark DF to Analysis DF using Arrow.
I have some cases in which I need to "materialize" Spark DF as fast as possible, do some local transformations, then distribute the payload again.
Based on description this can work pretty nice with Arrow batches between so the overhead can be minimized.

ideally a utility function would look like this:
var fxDataFrame = sparkDataFrame.ToFxDF();

and maybe reversed function as well.

var sparkDataFrame= fxDataFrame .ToSparkDF();

Any thoughts ?

Thanks,
Aleks

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.