dotnet / dotnet/spark

[FEATURE REQUEST]: Consider making the execution of UDF lambda functions more performant

Open
#288 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

This issue is sort of related to #284. Basically the problem here is that the current way in which Spark.NET runs UDFs causes the execution of them over a lot of data to be very costly performance-wise. I'm not sure if I have understood 100% how this works (correct me if I'm wrong) but I believe that the workflow is like:
1. Two processes are running at the same time always: The Java application which is controlling Spark, and the .NET application which is the one using Spark.NET to communicate through a socket to the Java application in order to perform Spark operations.
2. When a UDF is invoked from the .NET application into the Java process, then the Java app creates a third application (a Spark Worker) which is another .NET application which is the one that will execute the UDF.
3. This Worker process uses reflection in order to load the main .NET application in order to wrap the UDF lambda and then executes it for a batch of all records.
4. Step 2 is repeated x number of times by batches until all rows have been processed through the udf.

Because of all this work, I am seeing very big perf differences when running code using UDFs in Scala, than when using Spark.NET (it takes 10+ times longer in Spark.NET). We can use this issue to try to figure out if there is a better way to execute lambda functions in order to improve performance.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.