linkedin / linkedin/dr-elephant
Setting drelephant.analysis.fetch.initial.windowMillis causes java.lang.NullPointerException at com.linkedin.drelephant.analysis.AnalyticJobGeneratorHadoop2.configure(AnalyticJobGeneratorHadoop2.java:117)
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.4k
- Forks
- 839
- PR merge metrics
- No merged PRs in 30d
Description
When using the v2.0.13 (https://github.com/linkedin/dr-elephant/releases) release uncommenting the below property from dr-elephant/app-conf/GeneralConf.xml causes a null pointer exception on startup - and analysis never starts.
drelephant.analysis.fetch.initial.windowMillis 3600000When this property is commented back out everything starts up fine and it starts working through the entire JobHistory (9600 or so queue taking forever, it is my understanding that this parameter is supposed to prompt Dr Elephant to only look at the last hour).
I traced the code down to the highlighted operation below. This operation is triggering a NullPointerException for some unknown reason to me.
AnalyticJobGeneratorHadoop2.java:117 -> https://github.com/linkedin/dr-elephant/blob/master/app/com/linkedin/drelephant/analysis/AnalyticJobGeneratorHadoop2.java
@Override
public void configure(Configuration configuration)
throws IOException {
this.configuration = configuration;
String initialFetchWindowString = configuration.get(FETCH_INITIAL_WINDOW_MS);
if (initialFetchWindowString != null) {
long initialFetchWindow = Long.getLong(initialFetchWindowString);
_lastTime = System.currentTimeMillis() - FETCH_DELAY - initialFetchWindow;
_fetchStartTime = _lastTime;
}
updateResourceManagerAddresses();
}
08-09-2017 12:31:47 ERROR [Thread-6] com.linkedin.drelephant.ElephantRunner : Error occurred when configuring the analysis provider.
java.lang.NullPointerException
at com.linkedin.drelephant.analysis.AnalyticJobGeneratorHadoop2.configure(AnalyticJobGeneratorHadoop2.java:117)
at com.linkedin.drelephant.ElephantRunner.loadAnalyticJobGenerator(ElephantRunner.java:86)
at com.linkedin.drelephant.ElephantRunner.access$100(ElephantRunner.java:50)
at com.linkedin.drelephant.ElephantRunner$1.run(ElephantRunner.java:103)
at com.linkedin.drelephant.ElephantRunner$1.run(ElephantRunner.java:98)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
at com.linkedin.drelephant.security.HadoopSecurity.doAs(HadoopSecurity.java:108)
at com.linkedin.drelephant.ElephantRunner.run(ElephantRunner.java:98)
at com.linkedin.drelephant.DrElephant.run(DrElephant.java:34)
08-09-2017 12:31:47 ERROR [Thread-6] com.linkedin.drelephant.ElephantRunner : java.lang.NullPointerException
08-09-2017 12:31:47 ERROR [Thread-6] com.linkedin.drelephant.ElephantRunner : java.lang.RuntimeException: java.lang.NullPointerException
at com.linkedin.drelephant.ElephantRunner.loadAnalyticJobGenerator(ElephantRunner.java:89)
at com.linkedin.drelephant.ElephantRunner.access$100(ElephantRunner.java:50)
at com.linkedin.drelephant.ElephantRunner$1.run(ElephantRunner.java:103)
at com.linkedin.drelephant.ElephantRunner$1.run(ElephantRunner.java:98)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
at com.linkedin.drelephant.security.HadoopSecurity.doAs(HadoopSecurity.java:108)
at com.linkedin.drelephant.ElephantRunner.run(ElephantRunner.java:98)
at com.linkedin.drelephant.DrElephant.run(DrElephant.java:34)
Caused by: java.lang.NullPointerException
at com.linkedin.drelephant.analysis.AnalyticJobGeneratorHadoop2.configure(AnalyticJobGeneratorHadoop2.java:117)
at com.linkedin.drelephant.ElephantRunner.loadAnalyticJobGenerator(ElephantRunner.java:86)
... 9 more
08-09-2017 12:32:26 INFO [Thread-6] com.linkedin.drelephant.ElephantRunner : Dr.elephant has started
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in app/com/linkedin/drelephant/analysis/AnalyticJobGeneratorHadoop2.java at configure(), especially line 117, and reproduce startup with drelephant.analysis.fetch.initial.windowMillis enabled. Trace the configuration value through this method and verify that startup completes without the reported NullPointerException and that fetching begins within the configured initial window.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100