[FEATURE REQUEST]: Consider adding support for common RDD operations
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 332
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 9
Description
I know that this issue has been closed as won't fix in the past (#101), but I just wanted to add some roadblocks that I've been hitting while porting some existing scala code into Spark.NET. For my scenario, we have some data input that we don't have a control off, and in order to use it for our processing, we need to make some transformations into the data (using map/reduce operations). We can't really persist these transformed version of the data, because it changes constantly, so we have to re-transform it again. In Scala we are able to make this transformations using the convenient RDD Apis that scala/spark provide like map/reduce, which are very fast. To give some context, it takes about 20 seconds to make these transformations on the input data (which is a 150MB file). Code also looks very simple, like:
```scala
var myDataFrame
.rdd
.map(r=>(((r.getInt(0), r.getInt(1)), (r.getInt(2), r.getString(3), r.getString(4)))))
.reduceByKey((r1, r2)=>if(r1._1>r2._1) r2 else r1)
.map(r=>(r._1._1, TreeMap(r._1._2->r._2)))
.reduceByKey(_ ++ _)
.collectAsMap
```
So without too much detail, what this is doing is transforming the data by merging the first two columns, and then reducing to get all the rows that have similar first two columns, and do some calculations on them, and then doing another grouping with more calculations. As I said this takes about 20 seconds in Scala. In order to get the exact same result in Spark.NET is much more complicated, as column transformation can't easily be done without using a UDF (as opposed to just calling map) and similar for the reduce operations, we require complex UDF logic using process-to-process comunication and Apache Arrow in order to get the same output. Because of the way that UDFs run in Spark.NET (meaning that Spark.NET code calls into Java, which spins off a worker process, which reflects into the UDF to finally execute it) these transformations are super costly. In the end, it takes about 20 minutes to run the same transformations when using Spark.NET.
Even if we don't want to bring all of the RDD support, could we consider adding map/reduce operations that could just run in proc on the Java side so we avoid all of the process-to-process overhead causing these operations to be super expensive? I also understand that this might not be trivial at all, given that what you pass in to the transform operations are lambdas, which we don't really have a good way of passing using our JVM Bridge, but have we thought of enabling scenarios like this one?
Contributor guide
Assessment
This issue has not been assessed yet.