microsoft / microsoft/SynapseML
Making synapse.ml compatible with pandas_on_pyspark
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
With pyspark 3.2.0 now supporting ```pandas_on_pyspark``` (earlier koalas), this would make things much easier for a lot of us travelling from the faraway world of pandas. Currently, we need to manipulate the data using ```pandas_on_pyspark``` API and whenever an interaction with synapse.ml (like ```featurizer```) is needed a necessary ```.to_spark``` and then ```.to_pandas_on_pyspark``` is needed. I currently have a context manager to do this and I wrap all my code that needs synapse.ml within this context_manager
Contributor guide
Research direction
Start by reproducing the reported workflow with pandas_on_pyspark and SynapseML interactions such as featurizer, noting where .to_spark and .to_pandas_on_pyspark are required. Review the existing context-manager approach described in the issue and define compatibility as using SynapseML without those conversions, then verify the workflow on pyspark 3.2.0.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python, scala, spark
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100