linkedin / linkedin/ml-ease

example failure on run

Open
#1 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
339
Forks
75
PR merge metrics
No merged PRs in 30d

Description

(1)
$hadoop version
Hadoop 0.20.203.0

$hadoop jar mlease-1.0-jar-with-dependencies.jar com.linkedin.mlease.regression.jobs.Regression sample-config.job
Exception in thread "main" java.lang.NoSuchMethodError: org.codehaus.jackson.JsonFactory.enable(Lorg/codehaus/jackson/JsonParser$Feature;)Lorg/codehaus/jackson/JsonFactory;
at org.apache.avro.Schema.(Schema.java:86)
at org.apache.avro.generic.GenericData.(GenericData.java:901)
at org.apache.avro.generic.GenericDatumReader.(GenericDatumReader.java:49)
at com.linkedin.mapred.AvroUtils.getSchemaFromFile(AvroUtils.java:143)
at com.linkedin.mapred.AvroUtils.getAvroInputSchema(AvroUtils.java:204)
at com.linkedin.mapred.AbstractAvroJob.createJobConf(AbstractAvroJob.java:283)
at com.linkedin.mlease.regression.jobs.Regression.run(Regression.java:41)
at com.linkedin.mlease.regression.jobs.Regression.main(Regression.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

(2)
$hadoop version
Hadoop 2.4.0

$hadoop jar mlease-1.0-jar-with-dependencies.jar com.linkedin.mlease.regression.jobs.Regression sample-config.job
14/10/11 09:58:40 INFO Configuration.deprecation: mapred.job.queue.name is deprecated. Instead, use mapreduce.job.queuename
14/10/11 09:58:40 INFO jobs.RegressionPrepare: Running the preparation job of admm with map.key = and num.blocks=20
14/10/11 09:58:40 INFO mapred.AvroUtils: Running hadoop job with input paths:
14/10/11 09:58:40 INFO mapred.AvroUtils: hdfs://ns1/user/junhui1/ml-ease/sample-data.avro
14/10/11 09:58:40 INFO mapred.AvroUtils: Output path=hdfs://ns1/user/junhui1/ml-ease/sample-out/tmp-data
14/10/11 09:58:40 INFO client.RMProxy: Connecting to ResourceManager at yz2135.hadoop.data.sina.com.cn/10.39.2.135:9031
14/10/11 09:58:41 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
14/10/11 09:58:43 INFO mapred.FileInputFormat: Total input paths to process : 1
14/10/11 09:58:44 INFO mapreduce.JobSubmitter: number of splits:2
14/10/11 09:58:44 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1406191544821_301459
14/10/11 09:58:45 INFO impl.YarnClientImpl: Submitted application application_1406191544821_301459
14/10/11 09:58:45 INFO mapreduce.Job: The url to track the job: http://yz2135.hadoop.data.sina.com.cn:9008/proxy/application_1406191544821_301459/
14/10/11 09:58:45 INFO mapreduce.Job: Running job: job_1406191544821_301459
14/10/11 09:58:52 INFO mapreduce.Job: Job job_1406191544821_301459 running in uber mode : false
14/10/11 09:58:52 INFO mapreduce.Job: map 0% reduce 0%
14/10/11 09:58:56 INFO mapreduce.Job: Task Id : attempt_1406191544821_301459_m_000000_0, Status : FAILED
Error: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
14/10/11 09:58:57 INFO mapreduce.Job: Task Id : attempt_1406191544821_301459_m_000001_0, Status : FAILED
Error: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
14/10/11 09:59:03 INFO mapreduce.Job: Task Id : attempt_1406191544821_301459_m_000001_1, Status : FAILED
Error: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
14/10/11 09:59:03 INFO mapreduce.Job: Task Id : attempt_1406191544821_301459_m_000000_1, Status : FAILED
Error: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
14/10/11 09:59:08 INFO mapreduce.Job: Task Id : attempt_1406191544821_301459_m_000000_2, Status : FAILED
Error: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
14/10/11 09:59:09 INFO mapreduce.Job: Task Id : attempt_1406191544821_301459_m_000001_2, Status : FAILED
Error: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
14/10/11 09:59:15 INFO mapreduce.Job: map 100% reduce 0%
14/10/11 09:59:15 INFO mapreduce.Job: Job job_1406191544821_301459 failed with state FAILED due to: Task failed task_1406191544821_301459_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

14/10/11 09:59:15 INFO mapreduce.Job: Counters: 13
Job Counters
Failed map tasks=7
Killed map tasks=1
Launched map tasks=8
Other local map tasks=6
Rack-local map tasks=2
Total time spent by all maps in occupied slots (ms)=68644
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=34322
Total vcore-seconds taken by all map tasks=34322
Total megabyte-seconds taken by all map tasks=52718592
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
14/10/11 09:59:15 INFO jobs.RegressionAdmmTrain: Now running Regression Train using ADMM...
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
at com.linkedin.mapred.AvroUtils.getAvroInputSchema(AvroUtils.java:204)
at com.linkedin.mapred.AbstractAvroJob.createJobConf(AbstractAvroJob.java:283)
at com.linkedin.mlease.regression.jobs.RegressionAdmmTrain.run(RegressionAdmmTrain.java:135)
at com.linkedin.mlease.regression.jobs.Regression.run(Regression.java:60)
at com.linkedin.mlease.regression.jobs.Regression.main(Regression.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the hadoop jar command with sample-config.job and inspect the dependency and runtime classpaths. Trace the reported failures through com.linkedin.mapred.AvroUtils, AbstractAvroJob, Regression, and RegressionAdmmTrain, especially the referenced lines. Done means the sample regression run completes without the reported linkage, task, or array-index failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.