Question: How to use DataFrame API to achieve the function equivalent to map/reduce in spark.net
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 332
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 9
Description
Hi,
we have a scenario that need to use the map/reduce function in spark.net,
For example, we want to call
```
public IEnumerable MapCallback(IEnumerable input)
{
// do something with `IEnumerable input`
}
df.Rdd.MapPartitions(MapCallback, true)
```
The thing is, we need this IEnumerable input so that we can do some operation on the row level.
In [Mobius](https://github.com/microsoft/Mobius), we can access all the Rdd-related APIs, but according to this [issue](https://github.com/dotnet/spark/issues/101) seems all the Rdd-related APIs are no longer accessible.
So we have the following questions:
1. Is there any API in current Spark.Net that can implement the function that exactly equivalent to Rdd.Map, Rdd.Reduce and other mapreduce related function? **Note that we need to deal with a IEnumerable with arbitrary number of elements in one row, i.e., we may not know how many elements (columns) in a row until runtime.**
2. If the answer of 1 is false, can we just download the source code, change the visibility of Rdd-related APIs to public, and build a private bits to use?
3. Any other related suggestions will be really appreciated.
Looking forward to your answer! Thanks a lot!
Contributor guide
Assessment
This issue has not been assessed yet.