UPDATE: Spark 2.x support in Dr. Elephant

Nobody has claimed this yet.

Hi everyone,

Sharing some updates from Linkedin.

We are successfully analyzing all Spark 2.x jobs using Dr. Elephant.

In the last couple of months, we have been trying hard to enable Spark support in Dr. Elephant but we faced several blockers due to the stability issues in the Spark History Server (SHS). We were finally able make it work at Linkedin after deploying a custom version of SHS which mostly builds on top of the pre-existing effort that is ongoing in the open source Spark branch to improve SHS. On top of it, we added a few patches to Spark that adds more metrics which we will be contributing to open source soon.

So, essentially, if you want Dr. Elephant to analyze Spark 2.x jobs, then you need this custom setup of Spark History Server, at least until all the work is part of an official release. Most of the open source SHS improvement work was done in SPARK-18085 (closed recently). From the Dr. Elephant perspective, all the related work is already part of the open source repository (master branch).

@edwinalu /@skakker will be sharing more details on the Spark SHS setup soon.

Regards,
Akshay

Start by reviewing the Spark History Server setup described in the issue and the related SPARK-18085 work. The issue says Dr. Elephant's Spark-related work is already in the open source master branch, while further setup details were to be shared; no specific code change or completion criteria are provided.

Written by the indexing model from the issue text.