opensearch-project / opensearch-project/data-prepper

Introduce configuration for purging least recently used models in anomaly detection plugin

Open
#3,293 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.
Currently (per release v2.4), anomaly detection processor offers cardinality keys support via identification_keys property. This property will create up to 5000 models per each key/value pair. After the limit is reached, the plugin will stop creating new models that can be observed by CardinalityOverflow metric.

Describe the solution you'd like
As an alternative to the existing solution, it would be beneficial to introduce a configuration where least recently used models are purged out and new ones are being automatically created. This way, the implementation becomes more dynamic for handling newly arriving cardinality keys.

Describe alternatives you've considered (Optional)
Other mechanisms for purging unused models can be considered.

Additional context
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the anomaly detection processor's identification_keys behavior and the CardinalityOverflow metric. Define how a least-recently-used purge configuration should interact with the existing 5000-model limit, and verify that newly arriving cardinality keys can create models after older models are removed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.