babysor / babysor/MockingBird

if np.isnan(grad_norm.cpu()): AttributeError: 'float' object has no attribute 'cpu'

Open
#160 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
36.9k
Forks
5.2k
PR merge metrics
No merged PRs in 30d

Description

运行环境:天池实验室notebook gpu
https://tianchi.aliyun.com/
https://dsw-dev.data.aliyun.com/
run:
!python synthesizer_train.py mandarin /data/nas/workspace/jupyter/data/SV2TTS/synthesizer

==========
err:
Arguments:
run_id: mandarin
syn_dir: /data/nas/workspace/jupyter/data/SV2TTS/synthesizer
models_dir: synthesizer/saved_models/
save_every: 1000
backup_every: 25000
force_restart: False
hparams:

Checkpoint path: synthesizer/saved_models/mandarin/mandarin.pt
Loading training data from: /data/nas/workspace/jupyter/data/SV2TTS/synthesizer/train.txt
Using model: Tacotron
[AMP WARNING][Frontend.cpp:121][1634655387:258952]Sleep 0.1s waiting for AMP Server socket: /tmp/harp/ccl_ipc_socket/system/server.socket
[AMP INFO][Frontend.cpp:152][1634655387:359287]pid=741, start to allocate gpu resource ...
Using device: cuda

Initialising Tacotron Model...

Trainable Parameters: 30.875M

Starting the training of Tacotron from scratch

Using inputs from:
/data/nas/workspace/jupyter/data/SV2TTS/synthesizer/train.txt
/data/nas/workspace/jupyter/data/SV2TTS/synthesizer/mels
/data/nas/workspace/jupyter/data/SV2TTS/synthesizer/embeds
Found 9845 samples
+----------------+------------+---------------+------------------+
| Steps with r=2 | Batch Size | Learning Rate | Outputs/Step (r) |
+----------------+------------+---------------+------------------+
| 20k Steps | 12 | 0.001 | 2 |
+----------------+------------+---------------+------------------+

Traceback (most recent call last):
File "synthesizer_train.py", line 35, in
train(**vars(args))
File "/data/nas/workspace/jupyter/MockingBird/synthesizer/train.py", line 200, in train
if np.isnan(grad_norm.cpu()):
AttributeError: 'float' object has no attribute 'cpu'

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with synthesizer/train.py around line 200 and reproduce the reported command in the GPU notebook environment. Trace the type of grad_norm at the np.isnan check; done means training proceeds past this check without the reported AttributeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.