opensearch-project / opensearch-project/security-analytics

[BUG] Threat Intel Monitor is not clubbing matched doc-ids into Finding for same IoC because of fanout

Open
#1,176 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
111
Forks
111
Avg merge
1d 17h
Merged PRs (30d)
13

Description

Detector findings are group of rules that a doc id matches. Since doc level monitor distributed approach guarantees that only one node in one execution of a monitor sees a given doc, a unique finding is guaranteed

But threat intel monitor creates findings per Ioc and links doc ids that contain that ioc. So every single node in distributed execution can create a finding for an ioc and link unique doc ids but since threat intel finding is identified by Ioc value and nto a doc id there would be duplicate findings.

Potential solutions:

  1. use locking mechanism to create finding. if finding exists, then update with doc ids

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing Threat Intel Monitor finding creation during distributed execution and how findings are identified by IoC value. Inspect the existing finding update or synchronization paths before choosing an approach. Done means concurrent nodes produce one finding per IoC containing all matching document IDs, with a regression test covering fanout.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.