opensearch-project / opensearch-project/data-prepper
Add persistence to Data Prepper Anomaly Detector Processor between service restarts
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Since the certainty of an Anomaly Detection model increases as more samples are available, it would be good if all the information that the Anomaly Detector plugin has calculated since it was started would be persisted on disk (or other mechanism) between service restarts, because that would improve model fidelity over time.
It would also be good to be able to configure a limit of historical information to be saved, per processor.
I think Data Prepper Anomaly Detector Processor is a big differential compared to other similar tools, it is an amazing feature
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names the Data Prepper Anomaly Detector Processor but no files or tests; begin by locating the processor and its current model-state handling. Clarify the persistence mechanism, restart behavior, and per-processor history limit, then define verification for retaining model information across service restarts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100