linkedin / linkedin/dr-elephant
Dr. Elephant can't access JobHistory server and ResourceManager if they configured to use HTTPS
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.4k
- Forks
- 839
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
I'm trying to set Dr.Elephant for our cluster. Here is a piece of configuration:
mapred-site.xml
```
mapreduce.jobhistory.http.policy
HTTPS_ONLY
mapreduce.jobhistory.webapp.https.address
HOST:PORT
```
yarn-site.xml
```
yarn.resourcemanager.ha.rm-ids
rm1,rm2
yarn.http.policy
HTTPS_ONLY
yarn.resourcemanager.webapp.address.rm1
HOST1:PORT
yarn.resourcemanager.webapp.address.rm2
HOST2:PORT
```
But classes like `com.linkedin.drelephant.analysis.AnalyticJobGeneratorHadoop2` and `com.linkedin.drelephant.mapreduce.fetchers.MapReduceFetcherHadoop2` ignores that these services might be using HTTPS and not HTTP. Does it make sense to add support for HTTPS?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading com.linkedin.drelephant.analysis.AnalyticJobGeneratorHadoop2 and com.linkedin.drelephant.mapreduce.fetchers.MapReduceFetcherHadoop2 to trace how JobHistory and ResourceManager URLs are built. Compare that handling with the HTTPS_ONLY and HA settings shown in mapred-site.xml and yarn-site.xml; done means both services can be accessed when configured for HTTPS.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- distributed-systems, security
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100