awslabs / awslabs/deequ

Rule Suggestions

Open
#435 3 comments 0 reactions 0 assignees View on GitHub
enhancement question
Dominant language
Scala
Stars
3.6k
Forks
586
Avg merge
13d 13h
Merged PRs (30d)
1

Description

@dariobig Thank you very much for adding this support. Have been planning to implement them at work.
Quick question: would it be feasible to add a new rule suggestion for exact count of the number of records, for a non-nullable column, as well ? (note: please do point me to the right API, in case this capability already exists)
Reasoning behind the ask: At the moment, we are getting aprroxCount which is way-off in some cases. In one particular case of mine, there is a difference of 5M records between the actual and approximation (for a non-nullable column with completeness as 1.0). This is happening for both total count and distinct values as well (planning to raise a bug for the same). This check will help during the data migration and replication scenarios (alongside others) where we can automatically determine the success of the migration activity.

_Originally posted by @krishna-chaitanya-meduri in https://github.com/awslabs/deequ/issues/434#issuecomment-1253256091_

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named in the issue. Start by locating Deequ’s rule-suggestion API and its existing approximate-count suggestions, then determine the expected exact-count behavior for non-nullable columns. Done means the API can propose the requested check and tests cover the total and distinct-count migration scenarios described.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.