awslabs / awslabs/python-deequ

Deequ/PyDeequ compatibility with Spark-Connect

Open
#221 10 comments 12 reactions 0 assignees View on GitHub
enhancement feature request
Dominant language
Jupyter Notebook
Stars
826
Forks
158
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
At the moment `python-deequ` relies on direct calls to the underlying Spark JVM via `SparkSession._jvm`. But in the PySpark Connect `_jvm` is not available at all. So, at the moment `python-deequ` works only PySpark Classic, not in PySpark Connect. There were already reported problems from some of Spark-Connect users: [reported incompatibility of python-deequ with Spark-Connect](https://github.com/awslabs/python-deequ/issues/192#issuecomment-2209498425).

**Describe the solution you'd like**
- Add into deequ-core a Spark-Connect plugin and the corresponding protobuf messages for deequ classes/objects;
- Add into `python-deequ` an alternative implementation of API that is built on top of generated from protobuf python classes.

**Describe alternatives you've considered**
At the moment I do not know any other alternative. Based on the documentation of Spark-Connect and discussions in Spark mailing list / Jira, there are no plans to support `_jvm` in Connect.

**Additional context**
It is not so hard to do. I [did it for educational purposes recently](https://semyonsinchenko.github.io/ssinchenko/post/porting_deequ_to_sparkconnect/) and it required:
- About 350 lines of protobuf code;
- About 550 lines of Scala code that implements plugin and protobuf parser;
- Corresponding updates of `python-deequ`.

I'm willing to make all the required work and support the protobuf code with a new Python API.

In the deequ-core it may be done as a submodule. In the python-deequ it may be done as an extras. For example, `pip install pydeequ` will work as today, but `pip install pydeequ[connect]` will install also some protobuf dependencies and generated from proto classes.

It seems to me that adopting of deequ/python-deequ to Spark-Connect may be done without breaking changes. And it opens not only a way to make pydeequ works on Spark-Connect, but also a potential for creating others deequ APIs for Spark-Connect Go, Spark-Connect Rust, etc. Also it opens a way to use deequ from Spark-Connect via Java/Scala.

P.S. Using protobuf allows also to decouple Scala from Python at all and avoid all the problems with `default`, `Option`, etc. And it may be used not only in Connect but in PySpark Classic too.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.