Cortex performance issues

Request Type

Bug

Work Environment

Problem Description

After updating Cortex and TheHive to the latest versions (respectively 3.0.0-1 and 3.4.0-1) I have noticed a drop in performance in the execution of Cortex analyzers, mainly when I launch several in block from Thehive.
In the Cortex log I receive the following message:

[info] o.t.c.s.ErrorHandler - GET /api/job/Xq1zYm0Bdo8Y_Io8dnA8/waitreport?atMost=1%20minute returned 500
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://application/user/audit#1151591280]] after [60000 ms]. Message of type [org.thp.cortex.services.AuditActor$Register]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.
        at akka.pattern.PromiseActorRef$.$anonfun$defaultOnTimeout$1(AskSupport.scala:635)
        at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:650)
        at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:874)
        at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:113)
        at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:107)
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:872)

And then the next one:

[error] o.e.d.DBConfiguration - ElasticSearch request failure: DELETE:/_search/scroll/?
StringEntity({"scroll_id":["DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAIrlFjdzaTEtXzY1UVgyZFN6SGdqTnBKVncAAAAAAACK5BY3c2kxLV82NVFYMmRTekhnak5wSlZ3AAAAAAAAiuYWN3NpMS1fNjVRWDJkU3pIZ2pOcEpWdwAAAAAAAIrnFjdzaTEtXzY1UVgyZFN6SGdqTnBKVncAAAAAAACK6BY3c2kxLV82NVFYMmRTekhnak5wSlZ3"]},Some(application/json))
 => ElasticError(404,404,None,None,None,List(),None)

I have noticed that this problem, I receive it when having activated in several analyzers the AutoExtract flag, when I deactivate it does not fail.
Also (I don't know if it has to see), when this happens the website is inaccessible and you can't navigate through it.

Steps to Reproduce

1. enable the auto extract flag in the Shodan and VirusTotal analyzers
2. Run this analyzers over several ips (120 in my case)
3. Wait
4. Try to access the website

in previous versions this worked like a lightning for me.

Thank you for all your efforts!!

Start by reproducing the batch run with AutoExtract enabled for Shodan and VirusTotal, using the reported 120-IP workload. Read the Cortex logs around the AuditActor AskTimeoutException and Elasticsearch DELETE scroll failure, then compare with AutoExtract disabled. Done means analyzer execution completes without the timeout and the website remains accessible.