awslabs / awslabs/python-deequ

Is PyDeequ 2.0 still being actively developed? State of PyDeequ in general

Open
#270 12 comments 0 reactions 0 assignees View on GitHub
needs-human question
Dominant language
Jupyter Notebook
Stars
826
Forks
158
Avg merge
9d 22h
Merged PRs (30d)
3

Description

We’re currently running PyDeequ on Spark 3.5 and planning a future migration to Spark 4.1 / Delta Lake 4.1 / Python 3.13. Before we commit further to PyDeequ, we’d like to understand the project’s current state.

The 2.0.0b1 beta was announced in January 2025 with Spark Connect support — great direction, since _jvm bridge calls aren’t supported under Spark Connect which is the default in Spark 4.x. However, the last PyPI release was v1.5 in April 2025 and there’s been no follow-up since.

Two questions:

1. Is the project in active development or maintenance mode?

2. Is there a plan to support Spark 4.1 and get 2.0 published to PyPI?

This will directly inform whether we continue building on PyDeequ or evaluate alternatives ahead of our migration. Thanks for any visibility.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.