awslabs / awslabs/deequ

[FEATURE] Supporing Aggregation metrics for a group

Open
#528 1 comment 0 reactions 0 assignees View on GitHub
analyzer enhancement
Dominant language
Scala
Stars
3.6k
Forks
586
Avg merge
13d 13h
Merged PRs (30d)
1

Description

**Is your feature request related to a problem? Please describe.**
We have a use-case where we need to generate aggregated metrics like SUM, Mean and scannable metrics like MAX, MIN, MIN-LENGHT, MAX-LENGTH on a group defined on a column (or columns) in dataframe.

**Describe the solution you'd like**
Currently, the ScanShareableFrequencyBasedAnalyzer has only CountDistinct, Distinctness, Entropy, Uniqueness and UniqueValueRatio implementation. I would like to extend similar implementation for all other scannable and aggregation metrics so that each metrics can be computed at group level.

**Describe alternatives you've considered**
- One option is that i externally run the groupBy clause on Dataframe and split the dataframe based on group. Later, iterate over it and then keep calling Analyszer on each group.
**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

Start with ScanShareableFrequencyBasedAnalyzer and its existing CountDistinct, Distinctness, Entropy, Uniqueness, and UniqueValueRatio implementations. Compare the requested SUM, Mean, MAX, MIN, MIN-LENGTH, and MAX-LENGTH behavior with the external groupBy approach; done means the requested aggregation and scannable metrics can be computed for each dataframe group.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.