DrE can't be deployed on existing clusters due to large backlog of jobs to be processed.

Nobody has claimed this yet.

Hi All - I'm currently working at Paypal and we're having an issue deploying Dr. Elephant. I'd like to start a discussion around the best solution and show what we've done to improve the speed of processing job data.

Currently, DrE is unable to process a large back log of jobs on a given cluster.

Related thread on the [mailing list](https://groups.google.com/forum/#!topic/dr-elephant-users/SZK8sRZ_KcI).

Increasing the number of threads being used to query and process responses from the job history server does not improve the speed of processing, and we can see an ever growing queue of jobs waiting to be processed by DrE.

The impact of increasing the number of executors to a large number (240-ish) is shown below. While the queue sized started going down, we noticed that it went back up and hovered around 10k as more jobs were submitted.

```
$ grep queue dr_elephant.log |tail -10

05-25-2016 17:37:15 INFO com.linkedin.drelephant.ElephantRunner : Job queue size is 9085
05-25-2016 17:38:15 INFO com.linkedin.drelephant.ElephantRunner : Job queue size is 9081
05-25-2016 17:39:15 INFO com.linkedin.drelephant.ElephantRunner : Job queue size is 9078
05-25-2016 17:40:14 INFO com.linkedin.drelephant.ElephantRunner : Job queue size is 9071
05-25-2016 17:41:15 INFO com.linkedin.drelephant.ElephantRunner : Job queue size is 9069
05-25-2016 17:42:15 INFO com.linkedin.drelephant.ElephantRunner : Job queue size is 9038
05-25-2016 17:43:15 INFO com.linkedin.drelephant.ElephantRunner : Job queue size is 9016
05-25-2016 17:44:15 INFO com.linkedin.drelephant.ElephantRunner : Job queue size is 9010
05-25-2016 17:45:15 INFO com.linkedin.drelephant.ElephantRunner : Job queue size is 8986
05-25-2016 17:46:15 INFO com.linkedin.drelephant.ElephantRunner : Job queue size is 8973
```

At that point we delved into JVisualVM to see where the majority of time was being spent:


The `readJsonNode` function handles both reading from the Job History Server and parsing the JSON.

The challenge to scalability becomes clear by looking at the network traffic. Each call to the job history server creates a separate TCP connection and gzip is not enabled by default.

Separate TCP Connections:

The number of TCP connections being created for each MR job is 4 + the number of tasks for that job.

Without Gzip for the request:

`GET /ws/v1/history/mapreduce/jobs/job_1464719949755_0001/conf`

With Gzip:

When you multiply this by many many jobs, the benefits from gzip become pretty large.

Patch + visualvm output due to patch forthcoming.

Start with ElephantRunner's queue-size processing and the readJsonNode function, then review the linked mailing-list thread and the JVisualVM and network evidence in this report. Reproduce the backlog against the Job History Server and work with maintainers to define completion, since the issue proposes a patch but gives no acceptance criteria.

Written by the indexing model from the issue text.