linkedin / linkedin/dr-elephant
Support for multiple Hadoop clusters
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.4k
- Forks
- 839
- PR merge metrics
- No merged PRs in 30d
Description
Currently, Dr. Elephant creates a `Configuration` object that's initialized using the Hadoop conf on the classpath. That means Dr. Elephant only talks to one RM and one NN. Many companies run multiple Hadoop clusters, and rather than provisioning one Dr. Elephant instance per Hadoop cluster, it would be nice if Dr. Elephant could support talking to multiple Hadoop clusters, using a separate configuration per cluster.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing where Dr. Elephant creates the Hadoop Configuration object and how the classpath Hadoop configuration selects the ResourceManager and NameNode. Determine how separate configurations could be represented and selected for multiple clusters. Done means one Dr. Elephant instance can communicate with multiple Hadoop clusters without relying on a single classpath configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100