使用 gpu retrain 分词模型时遇到训练停滞
- Dominant language
- C++
- Stars
- 4k
- Forks
- 592
- PR merge metrics
- No merged PRs in 30d
Description
根据readme说明安装了paddle-gpu
```
Python 3.6.8 (default, Apr 25 2019, 21:02:35)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import paddle.fluid
>>> paddle.fluid.install_check.run_check()
Running Verify Fluid Program ...
W0716 19:33:06.646426 13717 device_context.cc:252] Please NOTE: device: 0, CUDA Capability: 52, Driver API Version: 10.0, Runtime API Version: 10.0
W0716 19:33:06.653287 13717 device_context.cc:260] device: 0, cuDNN Version: 7.6.
Your Paddle Fluid works well on SINGLE GPU or CPU.
W0716 19:33:15.945983 13717 fuse_all_reduce_op_pass.cc:74] Find all_reduce operators: 2. To make the speed faster, some all_reduce ops are fused during training, after fusion, the number of all_reduce ops is 1.
Your Paddle Fluid works well on MUTIPLE GPU or CPU.
Your Paddle Fluid is installed successfully! Let's start deep Learning with Paddle Fluid now
```
在retrain模型的时候,总会在某处卡住,无法继续训练,没有报错。
使用 cpu retrain 时则没有这样的问题。
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the README retraining instructions and reproduce the tokenizer-model retraining on GPU and CPU using the reported Python 3.6.8 and paddle-gpu environment. Check where GPU training stops and capture the surrounding output, then compare it with the successful CPU run. Done means identifying a reproducible cause or a documented environment limitation and confirming the retraining completes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100