microsoft / microsoft/SynapseML
Adapt SynapseML for the new client and server interface of Spark
@mhamilton723 is already working on this.
Since Oct 20, 2022.
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
**Is your feature request related to a problem? Please describe.**
There is ongoing discussion on a new client and server interface for Spark [SPARK-39375](https://issues.apache.org/jira/browse/SPARK-39375), this future interface will default to DataFrame-like operations and not RDDs.
This might impact users of SynapseML because the library relies internally on RDD operations and it is probably a good idea to get involved on the upstream Apache Spark discussion to sort out the requirements SynapseML might need that are not yet available in the DataFrame API.
**Describe the solution you'd like**
Refactor SynapseML to make it compatible with Spark's new recommended practices.
**Additional context**
Some reference links:
https://issues.apache.org/jira/browse/SPARK-39375
and the detailed document:
https://docs.google.com/document/d/1Mnl6jmGszixLW4KcJU5j9IgpG9-UabS0dcM6PM2XGDc/edit#
AB#1979660
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.