Proposal: add Spark 4.1 support

Motivation

PyDeequ currently supports the Spark 3.5 line, while the upstream Deequ project publishes a compatible Spark 4.1 artifact: com.amazon.deequ:deequ:2.0.18-spark-4.1. Supporting this artifact would let PyDeequ run on Spark 4.1 while retaining Spark 3.5 support.

This is intended as generic Apache Spark support; it does not add platform-specific configuration or documentation.

Relationship to #283

This proposal is designed to follow the Spark 3.5 upgrade in #283. The intended final mapping would be:

{
    "3.5": "com.amazon.deequ:deequ:2.0.21-spark-3.5",
    "4.1": "com.amazon.deequ:deequ:2.0.18-spark-4.1",
}

I would coordinate rebasing/merge order with #283 to avoid overlapping changes in dependency metadata, CI, and documentation.

Proposed design

1. Add an exact SPARK_VERSION mapping for 4.1 to deequ:2.0.18-spark-4.1.
2. Expand the optional PySpark dependency range to allow the Spark 4.1 line.
3. Make the Py4J/Scala collection bridge work with both Scala 2.12 (Spark 3.5) and Scala 2.13 (Spark 4.1):
   • use scala.collection.JavaConverters, which is available in both lines;
   • create empty Scala sequences through the existing sequence-conversion helper rather than calling Seq.empty() through Py4J.
4. Add Spark 4.1 CI coverage with a compatible Python/Java runtime.
5. Add focused configuration and runtime tests, plus documentation for selecting Spark 4.1 via SPARK_VERSION=4.1.
6. Update package constraints and lock data so a Spark 4.1 installation receives PySpark-compatible pandas and NumPy versions.

Compatibility expectations

• A process selects one Deequ artifact based on its Spark runtime; it does not load Spark 3.5 and 4.1 artifacts together.
• Spark 3.5 behavior remains supported and continues to select its matching Deequ artifact.

Feedback requested

Would maintainers prefer this as a follow-up PR after #283 merges, or as a coordinated PR that incorporates/rebases onto #283's Spark 3.5 changes?

まず現在の SPARK_VERSION マッピングと #283 の Spark 3.5 変更を確認します。次に、オプションの PySpark 制約、Py4J/Scala コレクションブリッジ、CI 設定、対象を絞ったランタイムテスト、ドキュメント、パッケージロックデータを調査します。Spark 4.1 が Deequ 2.0.18 を選択し、Spark 3.5 のサポートが維持され、CI が成功し、ドキュメントに記載された設定が機能すれば完了です。