Feature request: pyspark 3.0.0 version support
Open
enhancement
- Dominant language
- Scala
- Stars
- 301
- Forks
- 129
- PR merge metrics
- No merged PRs in 30d
Description
Please fill out the form below.
### System Information
- **Spark or PySpark**:
- **SDK Version**:
- **Spark Version**:
- **Algorithm (e.g. KMeans)**:
### Describe the problem
When is pyspark 3.0.0 version support coming? Can we have an estimate to plan update in projects? Right now sagemaker-pyspark lib is not working with latest pyspark as expected.
- **Exact command to reproduce**:
Contributor guide
Research direction
No source files, tests, or entry points are identified in the issue. Start by reproducing the reported incompatibility with PySpark 3.0.0 and compare it with the supported Spark and SDK versions. Done means sagemaker-pyspark works with PySpark 3.0.0 and has verification for the compatibility change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, spark
- Domain
- data-engineering, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100