alibaba / alibaba/x-deeplearning
xdl_submit.py --config=config.json提交报FAILED
- Dominant language
- PureBasic
- Stars
- 4.3k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
hadoop bin /home/hadoop/hadoop-3.1.1/bin/hadoop
CMD: /home/hadoop/hadoop-3.1.1/bin/hadoop jar /usr/bin/xdl-yarn-scheduler-1.0.0-SNAPSHOT-jar-with-dependencies.jar com.alibaba.xdl.Client -c=config.json -f=/usr/bin/xdl-yarn-scheduler-1.0.0-SNAPSHOT-jar-with-dependencies.jar -uuid=4911031c-b68a-4bd8-9b27-747956171b4e
2019-01-22 20:27:53,067 INFO xdl.Client: Yarn client start success.
2019-01-22 20:27:53,162 INFO xdl.Client: Create application with id:[application_1547717863709_0021] success.
2019-01-22 20:27:53,893 INFO xdl.Utils: Path:[hdfs://searchns1/user/admin/.xdl/application_1547717863709_0021] not exists, create success.
2019-01-22 20:27:53,893 INFO xdl.Client: Application base path:[hdfs://searchns1/user/admin/.xdl/application_1547717863709_0021/].
2019-01-22 20:27:54,163 INFO xdl.Client: Upload file config.json to hdfs:/searchns1/user/admin/.xdl/application_1547717863709_0021/config.json success.
2019-01-22 20:27:54,163 INFO xdl.Client: begin to upload files to hdfs
2019-01-22 20:27:54,242 INFO xdl.Client: Upload file /usr/bin/xdl-yarn-scheduler-1.0.0-SNAPSHOT-jar-with-dependencies.jar to hdfs:/searchns1/user/admin/.xdl/application_1547717863709_0021/xdl-yarn-scheduler-1.0.0-SNAPSHOT-jar-with-dependencies.jar success.
2019-01-22 20:28:12,630 INFO xdl.Utils: Run cmd [tar -czf /tmp/xdl_local/4911031c-b68a-4bd8-9b27-747956171b4e/songyang31.tar.gz -C /home/admin ./songyang31] success.
2019-01-22 20:28:13,426 INFO xdl.Client: Upload file /home/admin/songyang31/ to hdfs://searchns1/user/admin/.xdl/application_1547717863709_0021/songyang31.tar.gz success.
2019-01-22 20:28:13,426 INFO xdl.Client: finish uploading files to hdfs
2019-01-22 20:28:13,426 INFO xdl.Client: Upload user files success.
2019-01-22 20:28:13,426 INFO xdl.Client: ApplicationMaster start command is: [$JAVA_HOME/bin/java -Xmx256M com.alibaba.xdl.AppMasterRunner -c=config.json -v=songyang31.tar.gz -u=admin -p=hdfs://searchns1/user/admin/.xdl/application_1547717863709_0021/ 1>/stdout 2>/stderr]
2019-01-22 20:28:13,535 INFO xdl.Client: local resources: {config.json=resource { scheme: "hdfs" host: "searchns1" port: -1 file: "/user/admin/.xdl/application_1547717863709_0021/config.json" } size: 485 timestamp: 1548160074127 type: FILE visibility: PUBLIC, xdl-yarn-scheduler-1.0.0-SNAPSHOT-jar-with-dependencies.jar=resource { scheme: "hdfs" host: "searchns1" port: -1 file: "/user/admin/.xdl/application_1547717863709_0021/xdl-yarn-scheduler-1.0.0-SNAPSHOT-jar-with-dependencies.jar" } size: 4784145 timestamp: 1548160074228 type: FILE visibility: PUBLIC}
2019-01-22 20:28:13,543 INFO xdl.Client: Master add CLASSPATH:$HADOOP_CONF_DIR
2019-01-22 20:28:13,544 INFO xdl.Client: Master add CLASSPATH:$HADOOP_COMMON_HOME/share/hadoop/common/*
2019-01-22 20:28:13,544 INFO xdl.Client: Master add CLASSPATH:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*
2019-01-22 20:28:13,544 INFO xdl.Client: Master add CLASSPATH:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*
2019-01-22 20:28:13,544 INFO xdl.Client: Master add CLASSPATH:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*
2019-01-22 20:28:13,544 INFO xdl.Client: Master add CLASSPATH:$HADOOP_YARN_HOME/share/hadoop/yarn/*
2019-01-22 20:28:13,544 INFO xdl.Client: Master add CLASSPATH:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
2019-01-22 20:28:13,544 INFO xdl.Client: Setup ApplicationMaster container success.
2019-01-22 20:28:13,567 INFO conf.Configuration: resource-types.xml not found
2019-01-22 20:28:13,567 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2019-01-22 20:28:13,572 INFO xdl.Client: Setup application context success.
2019-01-22 20:28:13,572 INFO xdl.Client: Submitting application application_1547717863709_0021
2019-01-22 20:28:13,830 INFO impl.YarnClientImpl: Submitted application application_1547717863709_0021
2019-01-22 20:28:13,832 INFO xdl.Client: AppMaster host N/A Start waiting application: application_1547717863709_0021 ends.
2019-01-22 20:28:17,186 INFO xdl.Client: Application application_1547717863709_0021 finish with state FINISHED
2019-01-22 20:28:17,189 INFO xdl.Utils: ================================FINAL STATUS==================================
2019-01-22 20:28:17,189 INFO xdl.Utils: application_1547717863709_0021 : FAILED
2019-01-22 20:28:17,189 INFO xdl.Utils: ================================FINAL STATUS==================================
2019-01-22 20:28:17,216 INFO xdl.Utils: Delete the hdfs dir:hdfs://searchns1/user/admin/.xdl/application_1547717863709_0021/ success.
1、以上报FAILED,请教是什么问题呢?
2、yarn logs -applicationId application_1547717863709_0021,出现以下信息,求解。
File /tmp/logs/admin/logs/application_1547717863709_0021 does not exist.
Can not find any log file matching the pattern: [ALL] for the application: application_1547717863709_0021
Can not find the logs for the application: application_1547717863709_0021 with the appOwner: admin
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the xdl_submit.py invocation and the logged Yarn application submission for application_1547717863709_0021. Investigate why the application reports FINISHED while the final status is FAILED, and why yarn logs cannot find the application log; done means identifying the failure cause and documenting a reproducible resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python
- Domain
- cli, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100