[FEATURE REQUEST]: Benchmark Spark.NET versus PySpark and SparkR

#### Motivation

Is it possible to replace the existing Spark.NET with one that takes Spark scalar/java **codes** or **Jars** and compile that to .NET using [IKVM](https://github.com/ikvm-revived/ikvm)?

@wwasabi
>I know this is outside your scope, CC you as community here could start investigating [ikvm](https://github.com/ikvm-revived/ikvm)

ChatGPT reply

PySpark is a Python API for Apache Spark which is a data processing framework. The Spark core is implemented by Scala and Java, but it also provides different wrappers including Python (PySpark), R (SparkR), and SQL (Spark SQL). [You can install Spark separately (which would include all of the wrappers), or install Python version only by using pip or conda](https://stackoverflow.com/questions/67222999/what-is-the-difference-between-pyspark-and-spark)[1](https://stackoverflow.com/questions/67222999/what-is-the-difference-between-pyspark-and-spark).

SparkR is an R package that provides a light-weight frontend to use Apache Spark from R. [It is similar to PySpark but for R users](https://stackoverflow.com/questions/67222999/what-is-the-difference-between-pyspark-and-spark)[1](https://stackoverflow.com/questions/67222999/what-is-the-difference-between-pyspark-and-spark).

[Spark.NET is a .NET library for Apache Spark which allows you to write Spark applications using .NET languages such as C# and F#](https://learn.microsoft.com/en-us/dotnet/spark/tutorials/get-started)[2](https://learn.microsoft.com/en-us/dotnet/spark/tutorials/get-started).

SparkR versus Sparklyr

SparkR is an official Spark library while sparklyr is created by the RStudio community[1](http://datasideoflife.com/?p=1403). [Due to the fact that currently Python is a favorite language for Data Scientists using Spark, Spark R libraries are evolving at a slower pace and in general catch-up with the functionality available in PySpark](http://datasideoflife.com/?p=1403)[1](http://datasideoflife.com/?p=1403).

sparklyr is an R package developed by RStudio folks and provides a complete dplyr backend to Spark, using the same dplyr syntax. That implies that switching between environments does not require changing of function names. [In contrast to SparkR, here we operate on tables/tibbles, which are mapped to Spark DataFrames](http://datasideoflife.com/?p=1403)[1](http://datasideoflife.com/?p=1403).

https://spark.rstudio.com/