Cortex website and API not responding to requests

Request Type

Bug

Work Environment

Problem Description

Cortex website and API not responding to requests.

Steps to Reproduce

A few minutes after (re)starting Cortex and a couple of jobs have been run, it is getting stuck, unresponsive or extremely slow.

Complementary information

thread-dump.txt

Workaround relaxing the situation a bit, but not permanently

The following akka configuration helped a little bit, but Cortex will still end of unresponsive after a day or so:

# Debugging and workaround for performance issues
akka {
  log-config-on-start = on

  actor {
    default-dispatcher {
      fork-join-executor {
        parallelism-max = 16
      }
      thread-pool-executor {
        fixed-pool-size = 16
      }
      throughput = 1
    }
    default-blocking-io-dispatcher {
      fork-join-executor {
        parallelism-max = 32
      }
      thread-pool-executor {
        fixed-pool-size = 32
      }
      throughput = 1
    }
  }
}

It seems like definitely having more blocking I/O dispatchers (whichever executor is used) than normal dispatchers is helping, but not solving with the issue.

Start with thread-dump.txt and the reported Cortex 3.1.1-1 environment, then reproduce the slowdown after restarting Cortex and running a few jobs. Compare behavior with the supplied Akka dispatcher configuration; done means the website and API remain responsive rather than becoming stuck or extremely slow after extended operation.