awslabs / awslabs/python-deequ
Proposal: add Spark 4.1 support
- Langage dominant
- Jupyter Notebook
- Étoiles
- 826
- Forks
- 158
- Merge moyen
- 9 j 22 h
- PR mergées (30 j)
- 3
Description
## Motivation
PyDeequ currently supports the Spark 3.5 line, while the upstream Deequ project publishes a compatible Spark 4.1 artifact: `com.amazon.deequ:deequ:2.0.18-spark-4.1`. Supporting this artifact would let PyDeequ run on Spark 4.1 while retaining Spark 3.5 support.
This is intended as generic Apache Spark support; it does not add platform-specific configuration or documentation.
## Relationship to #283
This proposal is designed to follow the Spark 3.5 upgrade in #283. The intended final mapping would be:
```python
{
"3.5": "com.amazon.deequ:deequ:2.0.21-spark-3.5",
"4.1": "com.amazon.deequ:deequ:2.0.18-spark-4.1",
}
```
I would coordinate rebasing/merge order with #283 to avoid overlapping changes in dependency metadata, CI, and documentation.
## Proposed design
1. Add an exact `SPARK_VERSION` mapping for `4.1` to `deequ:2.0.18-spark-4.1`.
2. Expand the optional PySpark dependency range to allow the Spark 4.1 line.
3. Make the Py4J/Scala collection bridge work with both Scala 2.12 (Spark 3.5) and Scala 2.13 (Spark 4.1):
- use `scala.collection.JavaConverters`, which is available in both lines;
- create empty Scala sequences through the existing sequence-conversion helper rather than calling `Seq.empty()` through Py4J.
4. Add Spark 4.1 CI coverage with a compatible Python/Java runtime.
5. Add focused configuration and runtime tests, plus documentation for selecting Spark 4.1 via `SPARK_VERSION=4.1`.
6. Update package constraints and lock data so a Spark 4.1 installation receives PySpark-compatible pandas and NumPy versions.
## Compatibility expectations
- A process selects one Deequ artifact based on its Spark runtime; it does not load Spark 3.5 and 4.1 artifacts together.
- Spark 3.5 behavior remains supported and continues to select its matching Deequ artifact.
## Feedback requested
Would maintainers prefer this as a follow-up PR after #283 merges, or as a coordinated PR that incorporates/rebases onto #283's Spark 3.5 changes?
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Commencez par examiner le mapping actuel de SPARK_VERSION et les changements de Spark 3.5 dans #283. Inspectez ensuite les contraintes PySpark optionnelles, le pont de collections Py4J/Scala, la configuration CI, les tests d’exécution ciblés, la documentation et les données des verrous de paquets. La tâche est terminée lorsque Spark 4.1 sélectionne Deequ 2.0.18, que Spark 3.5 reste pris en charge, que la CI passe et que la configuration documentée fonctionne.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- python, scala, spark
- Domaine
- build-system, data-engineering, distributed-systems, documentation, testing
- Type d'issue
- Fonctionnalité
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- Active
- Clarté
- Plutôt claire
- Accessibilité débutants
- 45/100