awslabs / awslabs/deequ

Extending MetricsRepository for ElasticSearch

Open
#232 3 comments 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
3.6k
Forks
586
Avg merge
13d 13h
Merged PRs (30d)
1

Description

I'd like to look at extending `MetricsRepository` to store metrics with elastic search. The idea would be to make it easy to graph metrics over time with Kibana. I found some difficulties doing this though and would appreciate any guidance:

**Issues with private AnalysisResult case class constructor**
* It required implementing the load method, which returns a `MetricsRepositoryMultipleResultsLoader`
* Implementing `MetricsRepositoryMultipleResultsLoader` requires a `get` method returning a Seq[AnalysisResult]. AnalysisResult constructor is private (because the object is private it seems I can't access apply method) so I can't construct instance of this

**Query on persisting of Analyzers**
* I also noticed Analyzers currently need to be stored, and the objects themselves are stored with java serialization to a String (I realise in file system implementation everything is stored as serialized java objects). However the only time this is used is when filtering for metrics you want to use (in Check.isNewestPointNonAnomalous). I wondered if it is necessary to persist the Analyzer objects or if we could instead persist some identifier for them so they can be looked up on that? It could make persisting a bit neater to not have the serialized objects stored. Just for ElasticSearch implementation I would have to either take a similar approach for persisting the Analyzers, or have some other way to serialize them to/from JSON

**Private Serialization methods**
* Just to note many of the methods in AnalysisResultSerde are also private, though some of those may be helpful when extending the MetricsRepository

Thoughts appreciated. I am happy to raise PRs for things if it's helpful

Contributor guide

Open the contributing guide

Research direction

Start by reading MetricsRepository, AnalysisResult, MetricsRepositoryMultipleResultsLoader, and AnalysisResultSerde, then inspect Check.isNewestPointNonAnomalous. The issue does not define a concrete API or acceptance test; first settle the repository and analyzer-persistence design, then define tests for the chosen behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.