awslabs / awslabs/python-deequ

Store incorrect data with a column where all the reasons for the failure

Open
#83 2 comments 1 reaction 0 assignees View on GitHub
enhancement question
Dominant language
Jupyter Notebook
Stars
826
Forks
158
Avg merge
9d 22h
Merged PRs (30d)
3

Description

I have a requirement to run Data Quality Test So I am using Amazon python-deequ for this. I am able to find the Data Quality Success/Failure Status, but next I want to get all the rows that failed with a column where all the reasons for the failure are logged and Store into another DataFrame/Hive Table.
Thanks

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are identified in the issue. Start by locating the Python API around data-quality check results and determine whether failed rows and failure reasons are exposed; done means returning those rows with a reason column and making them storable in another DataFrame or Hive table.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.