aws / aws/sagemaker-spark-container
Support Spark 4
- Dominant language
- Python
- Stars
- 43
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
Spark 4.0.0 was released on May 23, 2025. The latest version as of writing this message is 4.2.0.
Are there any plans to support the current major version of Spark?
The existing images offered in this repo are problematic, because:
- PySpark 3 does not officially support Python 3.12
- Python 3.9 is officially end of life and no longer supported
So none of the images in this repo are actually officially supported.
Contributor guide
Research direction
Start by reviewing the repository's existing Spark image definitions and the Python/PySpark version configuration. Check how the current images are built and which versions they support. Done means providing an image for Spark 4 with Python and PySpark versions that are officially supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100