使用node2vec_randomwallk的结果测试 cluster/Word2vecExample.scala,OOM
- Dominant language
- Java
- Stars
- 6.8k
- Forks
- 1.6k
- Avg merge
- 44m
- Merged PRs (30d)
- 1
Description
测试cluster/cluster/Word2vecExample.scala,使用的数据是基于ogbn-100M的随机游走,该数据集大约有1.1亿个节点;
游走的参数是:epoch=1, step=10,即每个节点随机游走一次,每次长度是10。
使用master分支的代码,测试spark-on-angel下的cluster/cluster/Word2vecExample.scala,
启动脚本如下:
source ./bin/spark-on-angel-env.sh
$SPARK_HOME/bin/spark-submit \
--master yarn-cluster\
--conf spark.ps.instances=6 \
--conf spark.ps.cores=4 \
--conf spark.ps.jars=$SONA_ANGEL_JARS \
--conf spark.ps.memory=10g \
--jars $SONA_SPARK_JARS \
--driver-memory 20g \
--num-executors 6 \
--verbose \
--executor-cores 4 \
--executor-memory 15g \
--conf spark.default.parallelism=100 \
--class com.tencent.angel.spark.examples.cluster.Word2vecExample \
--conf spark.hadoop.fs.viewfs.inner.cache=false \
lib/spark-on-angel-examples-3.2.0.jar \
input:$input output:$output embedding:32 negative:3 epoch:1 saveModelInterval:1 stepSize:0.01 batchSize:50 psPartitionNum:500 dataPartitionNum:120 remapping:false window:5 saveContextEmbedding:true
日志中显示的参数:
driver日志最大id:
param embedding = 32
param negative = 3
param epoch = 1
param saveModelInterval = 1
param stepSize = 0.01
param batchSize = 50
param psPartitionNum = 500
param dataPartitionNum = 120
param remapping = false
param window = 5
param saveContextEmbedding = true
dataPartitionNum=120
numDocs=111059956 minWordId=0 maxWordId=111059956 numTokens=1110599560 maxLength=10
但是一直有oom的报错
21/11/24 17:14:47 INFO RunningContext: =====================Server running context start=======================
21/11/24 17:14:47 INFO RunningContext: state = IDLE
21/11/24 17:14:47 INFO RunningContext: totalRunningRPCCounter = 1
21/11/24 17:14:47 INFO RunningContext: infligtingRPCCounter = 0
21/11/24 17:14:47 INFO RunningContext: oomCounter = 0
21/11/24 17:14:47 INFO RunningContext: maxRunningRPCCounter = 10000
21/11/24 17:14:47 INFO RunningContext: generalRunningRPCCounter = 6000
21/11/24 17:14:47 INFO RunningContext: lastOOMRunningRPCCounter = 0
21/11/24 17:14:47 INFO RunningContext: totalRPCCounter = 110
21/11/24 17:14:47 INFO RunningContext: total=1092
21/11/24 17:14:47 INFO RunningContext: normal=1075
21/11/24 17:14:47 INFO RunningContext: network=0
21/11/24 17:14:47 INFO RunningContext: channelInUseCounter=0
21/11/24 17:14:47 INFO RunningContext: oom=0
21/11/24 17:14:47 INFO RunningContext: unknown=0
21/11/24 17:14:47 INFO RunningContext: =====================Server running context end =======================
21/11/24 17:14:47 INFO RunningContext: =====================Server running context start=======================
21/11/24 17:14:47 INFO RunningContext: state = IDLE
21/11/24 17:14:47 INFO RunningContext: totalRunningRPCCounter = 2
21/11/24 17:14:47 INFO RunningContext: infligtingRPCCounter = 0
21/11/24 17:14:47 INFO RunningContext: oomCounter = 0
21/11/24 17:14:47 INFO RunningContext: maxRunningRPCCounter = 10000
21/11/24 17:14:47 INFO RunningContext: generalRunningRPCCounter = 6000
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "heartbeatThread"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-3"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RPCWorker-thread-6"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "heartbeatThread"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "nioEventLoopGroup-29-5"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "heartbeatThread"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "client-heartbeat"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "dispatcher-event-loop-55"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "dispatcher-event-loop-48"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RPCResponser-thread-5"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RPCResponser-thread-9"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RPCResponser-thread-8"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RPCResponser-thread-10"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "dispatcher-event-loop-0"
Contributor guide
Assessment
This issue has not been assessed yet.