TheHive-Project / TheHive-Project/Cortex

[BUG] Cortex is unresponsive if too much jobs

Open
#374 22 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Scala
Stars
1.6k
Forks
264
PR merge metrics
No merged PRs in 30d

Description

Request Type

Bug

Work Environment
Question Answer
OS version (server) Docker SWARM
OS version (client) Viperr,
Virtualized Env. True
Dedicated RAM 16 GB
vCPU 8
Cortex version / git hash 3.1.1
Package Type RPM, DEB, Docker, Binary, From source
Index type Elasticsearch
Attachments storage Local (GlusterFS)
Browser type & version Firefox and Chrome
Problem Description

When ending a big quantity of artefact to Cortex to get analyze by a few analyzers, cortex becam unresponsive. Front Page is blank while answering code 200 and it is impossible to get access or communication using API. At the end of all jobs that continue running, service is again available.

Issue is report are not sent back to thehive, you need to rerun analyzer and result is given directly (cached result)

Steps to Reproduce
  1. Add some artifact
  2. Run them to a big quantity of Analyzer
  3. Observate the unresponsivness
Possible Solutions

I did add this to the application.conf this helped in some case but not all.


akka {
  log-config-on-start = on

  actor {
    default-dispatcher {
      fork-join-executor {
        parallelism-max = 16
      }
      thread-pool-executor {
        fixed-pool-size = 16
      }
      throughput = 1
    }
    default-blocking-io-dispatcher {
      fork-join-executor {
        parallelism-max = 32
      }
      thread-pool-executor {
        fixed-pool-size = 32
      }
      throughput = 1
    }
  }
}

Complementary information

(add anything that can help identifying the problem such as log excerpts, screenshots, configuration dumps etc.)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with a large batch of analyzer jobs and inspect the service logs alongside application.conf. Trace the analyzer-job processing and API/front-page availability while jobs run, including whether reports are returned to TheHive. Done means Cortex remains responsive under the reproduced load and reports are delivered without requiring a rerun.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, elasticsearch, scala
Domain
api, backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.