Are there any plans to have this fetch jobs from AWS/EMR?

I could see a pretty compelling use case for running Dr Elephant on top of AWS to analyze Spark/MR jobs running on EMR.

What I have in mind is, Dr Elephant would be installed in a stand-alone box, and periodically poll the AWS API to check for alive clusters via DescribeCluster, get the hostname for each of those, and automatically fetch jobs running on each cluster and analyze them. The idea is when you have lots of short-running EMR clusters, you can have 1 centralized location with all the results. Optionally, maybe integrate with AWS Data Pipeline to figure out workflows.

Right now the way we have it setup is to run Dr Elephant on each EMR cluster, but this is far from ideal because we lose the results once the cluster goes down unless we export it, and have to reinstall it on every new cluster. It still works because we can have it running in a long-standing staging environment and make sure things are green before they go to prod. But in order to identify trends over multiple days this breaks down.

I haven't dug into the code yet, but what do you think about this idea? I believe there is currently no way to do such a thing, but have you ever had this request or is it something you would be open to consider in Dr Elephant? Happy to help contributing to that once I start looking at the code if you think it would be valuable.

The proposal describes a standalone Dr. Elephant instance polling AWS DescribeCluster for live EMR clusters, resolving their hostnames, and fetching jobs for analysis. Start by tracing the existing cluster and job-fetching paths; done would mean a concrete design and implementation plan for centralized results, with AWS Data Pipeline integration treated as optional.