Rule Suggestions
- Dominant language
- Scala
- Stars
- 3.6k
- Forks
- 586
- Avg merge
- 13d 13h
- Merged PRs (30d)
- 1
Description
@dariobig Thank you very much for adding this support. Have been planning to implement them at work.
Quick question: would it be feasible to add a new rule suggestion for exact count of the number of records, for a non-nullable column, as well ? (note: please do point me to the right API, in case this capability already exists)
Reasoning behind the ask: At the moment, we are getting aprroxCount which is way-off in some cases. In one particular case of mine, there is a difference of 5M records between the actual and approximation (for a non-nullable column with completeness as 1.0). This is happening for both total count and distinct values as well (planning to raise a bug for the same). This check will help during the data migration and replication scenarios (alongside others) where we can automatically determine the success of the migration activity.
_Originally posted by @krishna-chaitanya-meduri in https://github.com/awslabs/deequ/issues/434#issuecomment-1253256091_
Contributor guide
Research direction
No files, tests, or entry points are named in the issue. Start by locating Deequ’s rule-suggestion API and its existing approximate-count suggestions, then determine the expected exact-count behavior for non-nullable columns. Done means the API can propose the requested check and tests cover the total and distinct-count migration scenarios described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100