Cortex - Excessive resource usage leading to crash

Hello all!

We have a Linux 16.04 headless server which has been running TheHive\Cortex for over a year now with the following specs: 4 CPU cores, 16gb RAM, 400gb storage

Software:

ckinsel@hiveapp1:~$ sudo dpkg -l | grep thehive
[sudo] password for ckinsel: 
ii  thehive                               3.3.1-1                                    all          Scalable, Open Source and Free Security Incident Response Solutions
ckinsel@hiveapp1:~$ sudo dpkg -l | grep cortex
ii  cortex                                2.1.3-1                                    all          Powerful Observable Analysis Engine
ckinsel@hiveapp1:~$ sudo dpkg -l | grep elasticsearch
hi  elasticsearch                         5.6.16                                     all          Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide' book can be found at https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html

Problem:
After running for a while (roughly once per 48 hrs) Cortex.service will randomly utilize all resources on the server (See screenshot). This causes thehive.service and cortex.service to report failure in Systemctl, and the webapp goes down.
Additionally, 'sudo systemctl restart cortex.service' hangs, and requires the VM to be reset to get it out of the stuck state.

Evidence:
TheHive and Cortex logs (/var/log/thehive/application.log and /var/log/cortex/application.log) both contain many Java OutOfMemory errors:
Thehive:

2019-08-10 12:20:41,983 [ERROR] from akka.actor.ActorSystemImpl in application-akka.actor.default-dispatcher-5 - Uncaught error from thread [application-akka.actor.default-dispatcher-20]: GC overhead limit exceeded, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[application]

java.lang.OutOfMemoryError: GC overhead limit exceeded

Cortex:

2019-08-10 08:11:15,922 [WARN] from org.elasticsearch.transport.TransportService in elasticsearch[_client_][transport_client_boss][T#2] - Received response for a request that has timed out, sent [47381ms] ago, timed out [26291ms] ago, action [cluster:monitor/nodes/liveness], node [{#transport#-1}{OP-6tmJnT--3isBtUjB4NA}{127.0.0.1}{127.0.0.1:9300}], id [39064]

2019-08-10 08:13:30,562 [INFO] from play.core.server.AkkaHttpServer in Thread-6 - Stopping server...

2019-08-10 08:14:16,555 [ERROR] from akka.actor.OneForOneStrategy in application-akka.actor.default-dispatcher-9 - head of empty list

java.util.NoSuchElementException: head of empty listsud

2019-08-10 08:51:53,099 [ERROR] from akka.actor.ActorSystemImpl in application-akka.actor.default-dispatcher-8 - Uncaught error from thread [application-akka.actor.default-dispatcher-9]: Java heap space, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[application]

java.lang.OutOfMemoryError: Java heap space

Systemctl:

ckinsel@hiveapp1:~$ sudo systemctl status thehive.service

[sudo] password for ckinsel: 

   thehive.service - TheHive

   Loaded: loaded (/usr/lib/systemd/system/thehive.service; enabled; vendor preset: enabled)

   Active: failed (Result: exit-code) since Sat 2019-08-10 12:20:49 EDT; 1 day 20h ago

     Docs: https://thehive-project.org

  Process: 1303 ExecStart=/opt/thehive/bin/thehive -Dconfig.file=/etc/thehive/application.conf -Dlogger.file=/etc/thehive/logback.xml -Dpidfile.path=/dev/null (code=exited, status=255)

 Main PID: 1303 (code=exited, status=255)



Aug 09 08:36:20 hiveapp1 systemd[1]: Started TheHive.

Aug 10 12:20:49 hiveapp1 systemd[1]: thehive.service: Main process exited, code=exited, status=255/n/a

Aug 10 12:20:49 hiveapp1 systemd[1]: thehive.service: Unit entered failed state.

Aug 10 12:20:49 hiveapp1 systemd[1]: thehive.service: Failed with result 'exit-code'.

ckinsel@hiveapp1:~$ sudo systemctl status cortex.service

  cortex.service - cortex

   Loaded: loaded (/etc/systemd/system/cortex.service; enabled; vendor preset: enabled)

   Active: active (running) since Fri 2019-08-09 08:36:20 EDT; 3 days ago

     Docs: https://thehive-project.org

 Main PID: 1311 (java)

    Tasks: 30

   Memory: 3.4G

      CPU: 22h 54min 58.281s

   CGroup: /system.slice/cortex.service

           └─1311 java -Duser.dir=/opt/cortex -Dconfig.file=/etc/cortex/application.conf -Dlogger.file=/etc/cortex/logback.xml -Dpidfile.path=/dev/null -cp /opt/cortex/lib/../conf/:/opt/cortex/lib/org.thehive-project.cortex-2.1.3-1-sans-externalized.jar:/opt/cortex/lib/



Aug 09 08:36:20 hiveapp1 systemd[1]: Started cortex.

Journalctl:

Aug 10 12:20:49 hiveapp1 systemd[1]: thehive.service: Main process exited, code=exited, status=255/n/a

Aug 10 12:20:49 hiveapp1 systemd[1]: thehive.service: Unit entered failed state.

Aug 10 12:20:49 hiveapp1 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=thehive comm="systemd" exe="/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'

Aug 10 12:20:49 hiveapp1 systemd[1]: thehive.service: Failed with result 'exit-code'.

Aug 10 12:20:49 hiveapp1 kernel: audit_printk_skb: 57 callbacks suppressed


Aug 10 12:20:49 hiveapp1 kernel: audit: type=1131 audit(1565454049.464:6787): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=thehive comm="systemd" exe="/lib/s

Recovery/Support
After rebooting the VM, all services start with no issues, the CPU rests below 3%, and RAM below 4gb.
This issue sounds like a potential memory leak, but I have not noticed it steadily climbing, and I don't think this explains the 100% CPU utilization. I'd consider our implementation pretty standard, but haven't found others reporting the same issue here or on github.
I cannot identify when exactly the issue started, as we've not (knowingly) applied changes to the server which would cause this behavior, and it has been ongoing for roughly a week.

At this point I'm not exactly sure where to look next for more logs, or for a source for the issue. If anyone has any ideas, please let me know!

Start with /var/log/thehive/application.log and /var/log/cortex/application.log, then compare the supplied systemctl and journalctl output around the failure. Trace the recurring resource exhaustion and JVM OutOfMemory errors to a reproducible cause; done means both services remain stable without the reported crashes.