awslabs / awslabs/deequ

Removal of a metric from a metric repository

Open
#359 2 comments 0 reactions 0 assignees View on GitHub
enhancement question
Dominant language
Scala
Stars
3.6k
Forks
586
Avg merge
13d 13h
Merged PRs (30d)
1

Description

Is it possible to remove a metric from a metric repository? I have had a look at the APIs and see no obvious way of doing this. Even if the ability exists to update a metric, i.e. I could add a status tag to its key and modified it to 'cancelled', but again I see no means to do that.

The use case is that I receive data in batches, and each is assessed re its data quality as it comes in, the metrics for each batch is saved during its ingestion. At some time in the future it has been decided that a particular batch of data needs to be removed from the system, this removal should also remove any metrics related to that batch in the metric repository.

Any recommendations on the best way to achieve this given the current APIs?

Many thanks.

Contributor guide

Open the contributing guide

Research direction

The issue names no source files or tests. Start by tracing the metric-repository APIs used to save batch metrics and checking whether update or deletion operations exist. Done means establishing a supported way to remove metrics associated with a discarded batch and covering the behavior with tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.