awslabs / awslabs/python-deequ

Proposal: add Spark 4.1 support

Geschlossen
#286 5 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
enhancement needs-human
Vorherrschende Sprache
Jupyter Notebook
Sterne
826
Forks
158
Ø Merge
9 T. 22 Std.
Gemergte PRs (30 T.)
3

Beschreibung

## Motivation

PyDeequ currently supports the Spark 3.5 line, while the upstream Deequ project publishes a compatible Spark 4.1 artifact: `com.amazon.deequ:deequ:2.0.18-spark-4.1`. Supporting this artifact would let PyDeequ run on Spark 4.1 while retaining Spark 3.5 support.

This is intended as generic Apache Spark support; it does not add platform-specific configuration or documentation.

## Relationship to #283

This proposal is designed to follow the Spark 3.5 upgrade in #283. The intended final mapping would be:

```python
{
"3.5": "com.amazon.deequ:deequ:2.0.21-spark-3.5",
"4.1": "com.amazon.deequ:deequ:2.0.18-spark-4.1",
}
```

I would coordinate rebasing/merge order with #283 to avoid overlapping changes in dependency metadata, CI, and documentation.

## Proposed design

1. Add an exact `SPARK_VERSION` mapping for `4.1` to `deequ:2.0.18-spark-4.1`.
2. Expand the optional PySpark dependency range to allow the Spark 4.1 line.
3. Make the Py4J/Scala collection bridge work with both Scala 2.12 (Spark 3.5) and Scala 2.13 (Spark 4.1):
- use `scala.collection.JavaConverters`, which is available in both lines;
- create empty Scala sequences through the existing sequence-conversion helper rather than calling `Seq.empty()` through Py4J.
4. Add Spark 4.1 CI coverage with a compatible Python/Java runtime.
5. Add focused configuration and runtime tests, plus documentation for selecting Spark 4.1 via `SPARK_VERSION=4.1`.
6. Update package constraints and lock data so a Spark 4.1 installation receives PySpark-compatible pandas and NumPy versions.

## Compatibility expectations

- A process selects one Deequ artifact based on its Spark runtime; it does not load Spark 3.5 and 4.1 artifacts together.
- Spark 3.5 behavior remains supported and continues to select its matching Deequ artifact.

## Feedback requested

Would maintainers prefer this as a follow-up PR after #283 merges, or as a coordinated PR that incorporates/rebases onto #283's Spark 3.5 changes?

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne mit der Überprüfung des aktuellen SPARK_VERSION-Mappings und der Änderungen für Spark 3.5 in #283. Untersuche anschließend die optionalen PySpark-Constraints, die Py4J/Scala-Collection-Bridge, die CI-Konfiguration, fokussierte Laufzeit-Tests, die Dokumentation und die Daten der Package-Locks. Als abgeschlossen gilt die Aufgabe, wenn Spark 4.1 Deequ 2.0.18 auswählt, Spark 3.5 weiterhin unterstützt wird, die CI erfolgreich durchläuft und die dokumentierte Konfiguration funktioniert.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python, scala, spark
Bereich
build-system, data-engineering, distributed-systems, documentation, testing
Issue-Typ
Feature
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Aktiv
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
45/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.