alibaba / alibaba/euler

分布式训练时出现错误

Open
#61 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.9k
Forks
553
PR merge metrics
No merged PRs in 30d

Description

当我进行分布式训练时,发现程序卡住了,并不往下进行,然后读取日志后发现错误:
I0314 16:34:07.407644 323537 remote_graph.cc:91] Initialize RemoteGraph, connect to server monitor: [localhost:2181, /opt/cloudera/parcels/CDH/lib/zookeeper/]
Environment variable CLASSPATH not set!
getJNIEnv: getGlobalJNIEnv failed
Environment variable CLASSPATH not set!
getJNIEnv: getGlobalJNIEnv failed

但是我进行了环境变量的配置:
export BASE_PATH=/opt/cloudera/parcels/CDH/lib
export HBASE_HOME=$BASE_PATH/hbase
export PATH=$HBASE_HOME/bin:$PATH
export HCAT_HOME=$BASE_PATH/hive-hcatalog
export KYLIN_HOME=/usr/local/kylin
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64/server:$LD_LIBRARY_PATH
export HADOOP_HOME=/opt/cloudera/parcels/CDH-5.11.2-1.cdh5.11.2.p0.4/lib/hadoop
export LIBRARY_PATH=$HADOOP_HOME/lib/native:$LIBRARY_PATH
export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native:$LD_LIBRARY_PATH
export CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath --glob):$CLASSPATH

这是我的分布式训练命令:
bash tf_euler/scripts/dist_tf_euler.sh --data_dir hdfs://Aquarius:8020/data/ppi --euler_zk_addr localhost:2181 --euler_zk_path /opt/cloudera/parcels/CDH/lib/zookeeper/ --max_id 56944 --feature_idx 1 --feature_dim 50 --label_idx 0 --label_dim 121 --model graphsage_supervised --mode train

请问是什么原因造成这种错误呢,应该如何解决呢

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with scripts/dist_tf_euler.sh and the remote_graph.cc:91 log location, then reproduce the distributed training command with the reported Hadoop, Java, and CLASSPATH environment. Trace where the JNI environment is initialized and compare the process environment with the exported variables. Done means the command proceeds past RemoteGraph initialization without the CLASSPATH or getJNIEnv errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.