Spark jobs not showing up on Dr Elephant UI

Nobody has claimed this yet.

hello, I am having a similar issue like a lot of others mentioned but none of those tickets helped me resolve my issue. My spark jobs won't show up on Dr. Elephant UI. I can only see MapReduce jobs. I went through this thread but could not figure out where to find dr elephant logs for the spark jobs? I am on EMR with Hadoop v 2.7.3, Spark 2.1.1. All the configs you mentioned above exist in my cluster. I can see the running spark job on the Resource Manager UI as well as spark history server once it's completed.

spark.yarn.historyServer.address ip-10-XX-XX-X.ec2.internal:18080
spark.eventLog.dir hdfs:///var/log/spark/apps
Here is how my dr elephant folder looks like:
drwxr-xr-x 2 ec2-user ec2-user 4096 Oct 24 16:29 app-conf
drwxr-xr-x 2 ec2-user ec2-user 4096 Oct 17 22:29 bin
drwxr-xr-x 3 ec2-user ec2-user 4096 Oct 17 22:29 conf
-rwxr-xr-x 1 ec2-user ec2-user 1199 Oct 24 16:30 dr.log
drwxr-xr-x 2 ec2-user ec2-user 16384 Oct 17 22:29 lib
drwxr-xr-x 2 ec2-user ec2-user 4096 Oct 24 16:31 logs
-rwxr-xr-x 1 ec2-user ec2-user 2925 Oct 17 22:26 README.md
-rw-r--r-- 1 root root 5 Oct 24 16:30 RUNNING_PID
drwxr-xr-x 3 ec2-user ec2-user 4096 Oct 17 22:29 scripts
drwxr-xr-x 3 ec2-user ec2-user 4096 Oct 17 22:29 share
echo $SPARK_HOME
/usr/lib/spark

echo $SPARK_CONF_DIR
/usr/lib/spark/conf
Am I missing something here? Please help.

thanks,
Kartik.

Start with dr.log and the logs directory, then review the app-conf and conf directories alongside README.md. Compare the Spark job configuration and ingestion path with the MapReduce path, using the Resource Manager and Spark history server to verify events; done means completed Spark jobs appear in the Dr. Elephant UI.

Written by the indexing model from the issue text.