linkedin / linkedin/dr-elephant

Setting drelephant.analysis.fetch.initial.windowMillis causes java.lang.NullPointerException at com.linkedin.drelephant.analysis.AnalyticJobGeneratorHadoop2.configure(AnalyticJobGeneratorHadoop2.java:117)

Open
#273 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
1.4k
Forks
839
PR merge metrics
No merged PRs in 30d

Description

When using the v2.0.13 (https://github.com/linkedin/dr-elephant/releases) release uncommenting the below property from dr-elephant/app-conf/GeneralConf.xml causes a null pointer exception on startup - and analysis never starts.

drelephant.analysis.fetch.initial.windowMillis 3600000

When this property is commented back out everything starts up fine and it starts working through the entire JobHistory (9600 or so queue taking forever, it is my understanding that this parameter is supposed to prompt Dr Elephant to only look at the last hour).

I traced the code down to the highlighted operation below. This operation is triggering a NullPointerException for some unknown reason to me.

AnalyticJobGeneratorHadoop2.java:117 -> https://github.com/linkedin/dr-elephant/blob/master/app/com/linkedin/drelephant/analysis/AnalyticJobGeneratorHadoop2.java

@Override
public void configure(Configuration configuration)
throws IOException {
this.configuration = configuration;
String initialFetchWindowString = configuration.get(FETCH_INITIAL_WINDOW_MS);
if (initialFetchWindowString != null) {
long initialFetchWindow = Long.getLong(initialFetchWindowString);
_lastTime = System.currentTimeMillis() - FETCH_DELAY - initialFetchWindow;
_fetchStartTime = _lastTime;
}
updateResourceManagerAddresses();
}

08-09-2017 12:31:47 ERROR [Thread-6] com.linkedin.drelephant.ElephantRunner : Error occurred when configuring the analysis provider.
java.lang.NullPointerException
at com.linkedin.drelephant.analysis.AnalyticJobGeneratorHadoop2.configure(AnalyticJobGeneratorHadoop2.java:117)
at com.linkedin.drelephant.ElephantRunner.loadAnalyticJobGenerator(ElephantRunner.java:86)
at com.linkedin.drelephant.ElephantRunner.access$100(ElephantRunner.java:50)
at com.linkedin.drelephant.ElephantRunner$1.run(ElephantRunner.java:103)
at com.linkedin.drelephant.ElephantRunner$1.run(ElephantRunner.java:98)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
at com.linkedin.drelephant.security.HadoopSecurity.doAs(HadoopSecurity.java:108)
at com.linkedin.drelephant.ElephantRunner.run(ElephantRunner.java:98)
at com.linkedin.drelephant.DrElephant.run(DrElephant.java:34)
08-09-2017 12:31:47 ERROR [Thread-6] com.linkedin.drelephant.ElephantRunner : java.lang.NullPointerException
08-09-2017 12:31:47 ERROR [Thread-6] com.linkedin.drelephant.ElephantRunner : java.lang.RuntimeException: java.lang.NullPointerException
at com.linkedin.drelephant.ElephantRunner.loadAnalyticJobGenerator(ElephantRunner.java:89)
at com.linkedin.drelephant.ElephantRunner.access$100(ElephantRunner.java:50)
at com.linkedin.drelephant.ElephantRunner$1.run(ElephantRunner.java:103)
at com.linkedin.drelephant.ElephantRunner$1.run(ElephantRunner.java:98)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
at com.linkedin.drelephant.security.HadoopSecurity.doAs(HadoopSecurity.java:108)
at com.linkedin.drelephant.ElephantRunner.run(ElephantRunner.java:98)
at com.linkedin.drelephant.DrElephant.run(DrElephant.java:34)
Caused by: java.lang.NullPointerException
at com.linkedin.drelephant.analysis.AnalyticJobGeneratorHadoop2.configure(AnalyticJobGeneratorHadoop2.java:117)
at com.linkedin.drelephant.ElephantRunner.loadAnalyticJobGenerator(ElephantRunner.java:86)
... 9 more

08-09-2017 12:32:26 INFO [Thread-6] com.linkedin.drelephant.ElephantRunner : Dr.elephant has started

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in app/com/linkedin/drelephant/analysis/AnalyticJobGeneratorHadoop2.java at configure(), especially line 117, and reproduce startup with drelephant.analysis.fetch.initial.windowMillis enabled. Trace the configuration value through this method and verify that startup completes without the reported NullPointerException and that fetching begins within the configured initial window.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.