2021Tencent Rhino-bird Open-source Training Program—Angel--刘倩

Open

#100 opened on Aug 5, 2021 0 comments 0 reactions 0 assignees View on GitHub

Dominant language: Scala
Stars: 173
Forks: 51
PR merge metrics: No merged PRs in 30d

Description

# 一、 angel 算法案例
### 1.1 LR-spark-on-angel输出
![image](https://user-images.githubusercontent.com/33976181/128314444-c98f06f1-1f04-417b-bbf0-203476d618f8.png)
![image](https://user-images.githubusercontent.com/33976181/128314475-3da47a8c-5f18-46c3-a5ee-7c5460dd2342.png)
![image](https://user-images.githubusercontent.com/33976181/128314496-f0f14a48-d0ba-424c-b9d8-5fb15d0b80e7.png)
![image](https://user-images.githubusercontent.com/33976181/128314517-115ca45a-f83d-4945-adc1-16c32e414582.png)
### 1.2 Debug
### 1. netty-all-4.1.1.Final.jar与json4s-jackson_2.11-3.4.2.jar版本问题
修改angel-ps与spark-on-angel的pom文件改为以上版本
### 2. 跑通项目的软件版本
apache-maven-3.8.1
hadoop-2.7.2
jdk1.8.0_161
protobuf-2.5.0
scala-2.11.8
spark-2.3.0-bin-hadoop2.7
angel-2.4.0-bin

# 二、 Pytorch on angel 算法案例
### 1.1 deepfm for torch on angel输出
http://hadoop001:8088/cluster/apps
![image](https://user-images.githubusercontent.com/33976181/128314609-a1b29ca7-d3aa-4e08-a538-7bfb3403a814.png)
![image](https://user-images.githubusercontent.com/33976181/128314670-ffebd5f3-164c-4d5b-bf98-e5db66692710.png)
![image](https://user-images.githubusercontent.com/33976181/128314685-ba79475d-7e63-4329-a070-624e1a4f86cb.png)
### 1.2 Debug
### 1. cmake报错
![image](https://user-images.githubusercontent.com/33976181/128314714-2b482abe-855d-4169-bfdf-c95a3b27dbe6.png)
在dockerfile里面添加
ENV Torch_DIR=/opt/libtorch/share/cmake/Torch
### 2. pytorch版本和torchvision版本不对应
在dokerfile文件里面添加torchvision=0.4.2
### 3. spark-submit提交脚本
source /home/liuqian/angel/angel/dist/target/angel-2.4.0-bin/bin/spark-on-angel-env.sh
### 4.内存问题
![image](https://user-images.githubusercontent.com/33976181/128314795-c1703eb1-2385-400a-9f1c-62ffcac7ab50.png)
把yarn.scheduler.capacity.maximum-am-resource-percent调到0.6
### 5.提交脚本内存分配不合理
![image](https://user-images.githubusercontent.com/33976181/128314854-c0df1f9c-3ebe-40b5-9ada-7dd3bd969265.png)
ps log
![image](https://user-images.githubusercontent.com/33976181/128314928-9e7548c4-a963-45a1-90b0-a374c1988bec.png)
换一台物理内存大的机器，重新配置跟之前一样的环境，yarn设置和提交脚本如下：
![image](https://user-images.githubusercontent.com/33976181/128314981-77855f96-e298-4e84-8a7b-16c63e9f11e0.png)
![image](https://user-images.githubusercontent.com/33976181/128315003-befe4313-440e-48d8-8da6-62e8bccc1eaa.png)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

2021Tencent Rhino-bird Open-source Training Program—Angel--刘倩

Description

Contributor guide

Assessment

Get new issues in your inbox