baidu / baidu/lac

使用 gpu retrain 分词模型时遇到训练停滞

Open
#100 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
4k
Forks
592
PR merge metrics
No merged PRs in 30d

Description

根据readme说明安装了paddle-gpu
```
Python 3.6.8 (default, Apr 25 2019, 21:02:35)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import paddle.fluid
>>> paddle.fluid.install_check.run_check()
Running Verify Fluid Program ...
W0716 19:33:06.646426 13717 device_context.cc:252] Please NOTE: device: 0, CUDA Capability: 52, Driver API Version: 10.0, Runtime API Version: 10.0
W0716 19:33:06.653287 13717 device_context.cc:260] device: 0, cuDNN Version: 7.6.
Your Paddle Fluid works well on SINGLE GPU or CPU.
W0716 19:33:15.945983 13717 fuse_all_reduce_op_pass.cc:74] Find all_reduce operators: 2. To make the speed faster, some all_reduce ops are fused during training, after fusion, the number of all_reduce ops is 1.
Your Paddle Fluid works well on MUTIPLE GPU or CPU.
Your Paddle Fluid is installed successfully! Let's start deep Learning with Paddle Fluid now
```
在retrain模型的时候,总会在某处卡住,无法继续训练,没有报错。
使用 cpu retrain 时则没有这样的问题。

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the README retraining instructions and reproduce the tokenizer-model retraining on GPU and CPU using the reported Python 3.6.8 and paddle-gpu environment. Check where GPU training stops and capture the surrounding output, then compare it with the successful CPU run. Done means identifying a reproducible cause or a documented environment limitation and confirming the retraining completes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.