[FEATURE] Supporing Aggregation metrics for a group
- Dominant language
- Scala
- Stars
- 3.6k
- Forks
- 586
- Avg merge
- 13d 13h
- Merged PRs (30d)
- 1
Description
**Is your feature request related to a problem? Please describe.**
We have a use-case where we need to generate aggregated metrics like SUM, Mean and scannable metrics like MAX, MIN, MIN-LENGHT, MAX-LENGTH on a group defined on a column (or columns) in dataframe.
**Describe the solution you'd like**
Currently, the ScanShareableFrequencyBasedAnalyzer has only CountDistinct, Distinctness, Entropy, Uniqueness and UniqueValueRatio implementation. I would like to extend similar implementation for all other scannable and aggregation metrics so that each metrics can be computed at group level.
**Describe alternatives you've considered**
- One option is that i externally run the groupBy clause on Dataframe and split the dataframe based on group. Later, iterate over it and then keep calling Analyszer on each group.
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Research direction
Start with ScanShareableFrequencyBasedAnalyzer and its existing CountDistinct, Distinctness, Entropy, Uniqueness, and UniqueValueRatio implementations. Compare the requested SUM, Mean, MAX, MIN, MIN-LENGTH, and MAX-LENGTH behavior with the external groupBy approach; done means the requested aggregation and scannable metrics can be computed for each dataframe group.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100