关于2.0.0 gdbt模型 较大数据量程序跑死导致的java.lang.InterruptedException问题。
- Linguagem predominante
- Java
- Estrelas
- 6.8k
- Forks
- 1.6k
- Merge médio
- 44min
- PRs com merge (30d)
- 1
Descrição
在进行小数据量(5000条)的gdbt模型训练可以正常输出训练结果,但是在20w条数据量时,程序会运行16个小时。某些worker报org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: Error reading the stream java.io.IOException: No such process异常。
训练200w条数据时报
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1252)
at java.lang.Thread.join(Thread.java:1326)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeResponder(DFSOutputStream.java:638)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeInternal(DFSOutputStream.java:606)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:602)
异常。
我想问是不是我ps的数量设的不对?
我的启动脚本如下:
epochNum=3
learnType=c
workerNumber=5
workerMemory=5
workerCpu=2
taskNumber=5
taskMemory=5
storageLevel=memory_disk
PSNumber=1
PSMemory=5
$ANGEL_HOME/bin/angel-submit \
--action.type train \
--angel.app.submit.class com.tencent.angel.ml.GBDT.GBDTRunner \
--angel.train.data.path $input_path \
--angel.save.model.path $model_path \
--angel.log.path $log_path \
--ml.data.type libsvm \
--ml.feature.index.range $featureNum \
--ml.feature.num $featureNum \
--ml.gbdt.tree.num 3 \
--ml.gbdt.tree.depth 3 \
--ml.gbdt.split.num 10 \
--ml.data.validate.ratio 0.1 \
--ml.gbdt.sample.ratio 1 \
--ml.epoch.num $epochNum \
--ml.learn.rate $learnRate \
--angel.workergroup.number $workerNumber \
--angel.worker.memory.gb $workerMemory \
--angel.worker.cpu.vcores $workerCpu \
--angel.worker.task.number $taskNumber \
--angel.task.data.storage.level $storageLevel \
--angel.task.memorystorage.max.gb $taskMemory \
--angel.ps.number $PSNumber \
--angel.ps.memory.gb $PSMemory \
--angel.am.memory.gb 4 \
--angel.staging.dir /user/weibo_bigdata_push/angel_stage \
--angel.tmp.output.path.prefix /user/weibo_bigdata_push/angel_stage
请问一下是ps数量或者memroy设小么?
为什么训练20w条数据还不够么?
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.