if np.isnan(grad_norm.cpu()): AttributeError: 'float' object has no attribute 'cpu'
- Dominant language
- Python
- Stars
- 36.9k
- Forks
- 5.2k
- PR merge metrics
- No merged PRs in 30d
Description
运行环境:天池实验室notebook gpu
https://tianchi.aliyun.com/
https://dsw-dev.data.aliyun.com/
run:
!python synthesizer_train.py mandarin /data/nas/workspace/jupyter/data/SV2TTS/synthesizer
==========
err:
Arguments:
run_id: mandarin
syn_dir: /data/nas/workspace/jupyter/data/SV2TTS/synthesizer
models_dir: synthesizer/saved_models/
save_every: 1000
backup_every: 25000
force_restart: False
hparams:
Checkpoint path: synthesizer/saved_models/mandarin/mandarin.pt
Loading training data from: /data/nas/workspace/jupyter/data/SV2TTS/synthesizer/train.txt
Using model: Tacotron
[AMP WARNING][Frontend.cpp:121][1634655387:258952]Sleep 0.1s waiting for AMP Server socket: /tmp/harp/ccl_ipc_socket/system/server.socket
[AMP INFO][Frontend.cpp:152][1634655387:359287]pid=741, start to allocate gpu resource ...
Using device: cuda
Initialising Tacotron Model...
Trainable Parameters: 30.875M
Starting the training of Tacotron from scratch
Using inputs from:
/data/nas/workspace/jupyter/data/SV2TTS/synthesizer/train.txt
/data/nas/workspace/jupyter/data/SV2TTS/synthesizer/mels
/data/nas/workspace/jupyter/data/SV2TTS/synthesizer/embeds
Found 9845 samples
+----------------+------------+---------------+------------------+
| Steps with r=2 | Batch Size | Learning Rate | Outputs/Step (r) |
+----------------+------------+---------------+------------------+
| 20k Steps | 12 | 0.001 | 2 |
+----------------+------------+---------------+------------------+
Traceback (most recent call last):
File "synthesizer_train.py", line 35, in
train(**vars(args))
File "/data/nas/workspace/jupyter/MockingBird/synthesizer/train.py", line 200, in train
if np.isnan(grad_norm.cpu()):
AttributeError: 'float' object has no attribute 'cpu'
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with synthesizer/train.py around line 200 and reproduce the reported command in the GPU notebook environment. Trace the type of grad_norm at the np.isnan check; done means training proceeds past this check without the reported AttributeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100