awslabs / awslabs/python-deequ

Proposal: add Spark 4.1 support

Fermée
#286 5 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
enhancement needs-human
Langage dominant
Jupyter Notebook
Étoiles
826
Forks
158
Merge moyen
9 j 22 h
PR mergées (30 j)
3

Description

## Motivation

PyDeequ currently supports the Spark 3.5 line, while the upstream Deequ project publishes a compatible Spark 4.1 artifact: `com.amazon.deequ:deequ:2.0.18-spark-4.1`. Supporting this artifact would let PyDeequ run on Spark 4.1 while retaining Spark 3.5 support.

This is intended as generic Apache Spark support; it does not add platform-specific configuration or documentation.

## Relationship to #283

This proposal is designed to follow the Spark 3.5 upgrade in #283. The intended final mapping would be:

```python
{
"3.5": "com.amazon.deequ:deequ:2.0.21-spark-3.5",
"4.1": "com.amazon.deequ:deequ:2.0.18-spark-4.1",
}
```

I would coordinate rebasing/merge order with #283 to avoid overlapping changes in dependency metadata, CI, and documentation.

## Proposed design

1. Add an exact `SPARK_VERSION` mapping for `4.1` to `deequ:2.0.18-spark-4.1`.
2. Expand the optional PySpark dependency range to allow the Spark 4.1 line.
3. Make the Py4J/Scala collection bridge work with both Scala 2.12 (Spark 3.5) and Scala 2.13 (Spark 4.1):
- use `scala.collection.JavaConverters`, which is available in both lines;
- create empty Scala sequences through the existing sequence-conversion helper rather than calling `Seq.empty()` through Py4J.
4. Add Spark 4.1 CI coverage with a compatible Python/Java runtime.
5. Add focused configuration and runtime tests, plus documentation for selecting Spark 4.1 via `SPARK_VERSION=4.1`.
6. Update package constraints and lock data so a Spark 4.1 installation receives PySpark-compatible pandas and NumPy versions.

## Compatibility expectations

- A process selects one Deequ artifact based on its Spark runtime; it does not load Spark 3.5 and 4.1 artifacts together.
- Spark 3.5 behavior remains supported and continues to select its matching Deequ artifact.

## Feedback requested

Would maintainers prefer this as a follow-up PR after #283 merges, or as a coordinated PR that incorporates/rebases onto #283's Spark 3.5 changes?

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Commencez par examiner le mapping actuel de SPARK_VERSION et les changements de Spark 3.5 dans #283. Inspectez ensuite les contraintes PySpark optionnelles, le pont de collections Py4J/Scala, la configuration CI, les tests d’exécution ciblés, la documentation et les données des verrous de paquets. La tâche est terminée lorsque Spark 4.1 sélectionne Deequ 2.0.18, que Spark 3.5 reste pris en charge, que la CI passe et que la configuration documentée fonctionne.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python, scala, spark
Domaine
build-system, data-engineering, distributed-systems, documentation, testing
Type d'issue
Fonctionnalité
Difficulté
4/5
Temps estimé
3-5 jours
Activité
Active
Clarté
Plutôt claire
Accessibilité débutants
45/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.