Dr-Elephant not fetching RUNNING spark application (only succeeded and failed applications are fetched)

Hi,

Dr-Elephant is only fetching completed applications (filtered by SUCCEEDED or FAILED status).
Our spark streaming applications are always RUNNING non-stop (except for weekly restarts).
We want to be able to analyze them and generate real time heuristics.

Why does dr-elephant exclude running application ?
Is there a way to include them when fetching jobs list?

More details:

• We are using SparkFetcher.

• Dr. Elephant gets list of only succeeded and failed applications from Yarn History Server API:

applicationscom.linkedin.drelephant.analysis.AnalyticJobGeneratorHadoop2 : The succeeded apps URL is https://{YARN_RM_HOST}/ws/v1/cluster/apps?finalStatus=SUCCEEDED&finishedTimeBegin=1598632623112&finishedTimeEnd=1598632683376
com.linkedin.drelephant.analysis.AnalyticJobGeneratorHadoop2 : The failed apps URL is https://{ YARN_RM_HOST}/ws/v1/cluster/apps?finalStatus=FAILED&state=FINISHED&finishedTimeBegin=1598632623112&finishedTimeEnd=1598632683376

• And then data are fetched from the Spark History server API:

com.linkedin.drelephant.spark.fetchers.SparkRestClient : calling REST API at http://{SHS_HOST}/api/v1/applications/application_xxxxxxxxxx_xxxxxx
08-28-2020 18:56:09 INFO  [ForkJoinPool-1-worker-3] com.linkedin.drelephant.spark.fetchers.SparkRestClient : creating SparkApplication by calling REST API at http://{SHS_HOST}/api/v1/applications/application_xxxxxxxxxx_xxxxxx/1/logs to get eventlogs

Running spark application are available in both YARN HS and Spark HS. I can retrieve log events by accessing http://{SHS_HOST}/api/v1/applications/application_xxxxxxxxxx_xxxxxx/1/logs

Thank you