opensearch-project / opensearch-project/security-analytics

Refactor Threat Intel Feeds Job Scheduler

Open
#787 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
111
Forks
111
Avg merge
1d 17h
Merged PRs (30d)
13

Description

Refactor threat intel feeds job scheduler to decouple it with creating and updating IOCs.

Part 1.
Refactor the transport layer into a service class that DetectorThreatIntel directly calls.

Part 2.

  • Renamed TIFJobUpdateService to ThreatIntelFeedIndexService
  • Renamed TIFJobParameter to TIFJobSchedulerMetadata
  • Refactored ThreatIntelFeedDataService and ThreatIntelFeedIndexService so that threat intel indices are created and deleted in the ThreatIntelFeedIndexService class but the data is parsed and saved into the index in the ThreatIntelFeedDataService class to provide better modularization.

More specifics to the refactoring
Methods moved from ThreatIntelFeedDataService to ThreatIntelFeedIndexService

  • getIndexMapping
  • createIndexIfNotExists
  • deleteThreatIntelDataIndex
  • createIndexIfNotExists

Methods moved from ThreatIntelFeedIndexService to ThreatIntelFeedDataService

  • updateJobSchedulerMetadataAsSucceeded
  • waitUntilAllShardsStarted

Refactored

  • createThreatIntelFeedData to createThreatIntelFeed in ThreatIntelFeedIndexService, refactored so groupedActionListener is done in data service where it uses the csvParser to parse and save feeds into the index. This method is the hook for ThreatIntelHighLevelHandler and TIFJobRunner to set up and create a TIF index as well as populating it with data in ThreatIntelFeedDataService

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Locate DetectorThreatIntel, ThreatIntelFeedDataService, ThreatIntelFeedIndexService, TIFJobSchedulerMetadata, ThreatIntelHighLevelHandler, and TIFJobRunner. Read the current scheduler and transport interactions first, then trace the listed methods and their callers. Done means the renamed services and metadata are wired correctly, with index lifecycle separated from feed parsing and persistence.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.