awslabs / awslabs/python-deequ

[Feature Request]DuckDB as another analytic engine for Deequ

Open
#128 5 comments 1 reaction 0 assignees View on GitHub
enhancement feature request
Dominant language
Jupyter Notebook
Stars
826
Forks
158
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
Today, PyDeequ is a PySpark binding for Deequ which is in Scala and Spark only. While it is a good fit for DEs, Spark is not a great fit for many DS use-cases who will have datasets fit in memory and do not want to setup Spark.

See initial ideas here https://youtu.be/fvKFOfaLwBA?t=1393 from @sscdotopen.

**Describe the solution you'd like**

As discussed in above video, it would be good to create the proper abstractions to support another analytic engine. DuckDB who has gained popularity recently can be another analytic engine. The designs need more thoughts/discussion.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.