microsoft / microsoft/SynapseML

Making synapse.ml compatible with pandas_on_pyspark

Open
#1,308 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

With pyspark 3.2.0 now supporting ```pandas_on_pyspark``` (earlier koalas), this would make things much easier for a lot of us travelling from the faraway world of pandas. Currently, we need to manipulate the data using ```pandas_on_pyspark``` API and whenever an interaction with synapse.ml (like ```featurizer```) is needed a necessary ```.to_spark``` and then ```.to_pandas_on_pyspark``` is needed. I currently have a context manager to do this and I wrap all my code that needs synapse.ml within this context_manager

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported workflow with pandas_on_pyspark and SynapseML interactions such as featurizer, noting where .to_spark and .to_pandas_on_pyspark are required. Review the existing context-manager approach described in the issue and define compatibility as using SynapseML without those conversions, then verify the workflow on pyspark 3.2.0.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python, scala, spark
Domain
data, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.