babysor / babysor/MockingBird

跑8天出现这样的错误

Open
#774 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
36.9k
Forks
5.2k
PR merge metrics
No merged PRs in 30d

Description

{| Epoch: 55/167 (496/1926) | Loss: 0.2899 | 0.92 steps/s | Step: 424k | }Input at step 424500: 可以接受外面超市送的生活物资吗
{| Epoch: 55/167 (996/1926) | Loss: 0.2907 | 0.93 steps/s | Step: 425k | }Traceback (most recent call last):
File "E:\ai_ai\MockingBird-main\MockingBird-main\venv\lib\site-packages\torch\serialization.py", line 379, in save
_save(obj, opened_zipfile, pickle_module, pickle_protocol)
File "E:\ai_ai\MockingBird-main\MockingBird-main\venv\lib\site-packages\torch\serialization.py", line 499, in _save
zip_file.write_record(name, storage.data_ptr(), num_bytes)
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "E:\ai_ai\MockingBird-main\MockingBird-main\synthesizer_train.py", line 37, in
train(**vars(args))
File "E:\ai_ai\MockingBird-main\MockingBird-main\synthesizer\train.py", line 234, in train
model.save(backup_fpath, optimizer)
File "E:\ai_ai\MockingBird-main\MockingBird-main\synthesizer\models\base.py", line 58, in save
torch.save({
File "E:\ai_ai\MockingBird-main\MockingBird-main\venv\lib\site-packages\torch\serialization.py", line 380, in save
return
File "E:\ai_ai\MockingBird-main\MockingBird-main\venv\lib\site-packages\torch\serialization.py", line 259, in __exit__
self.file_like.write_end_of_file()
RuntimeError: [enforce fail at ..\caffe2\serialize\inline_container.cc:298] . unexpected pos 433112704 vs 433112592

Process finished with exit code -1073740791 (0xC0000409)

Contributor guide

No contributing guide indexed for this repository

Research direction

The traceback reaches synthesizer/train.py:234 and synthesizer/models/base.py:58; start there and inspect the checkpoint path and available storage during torch.save. Reproduce the long-running training save, then confirm that a checkpoint completes without the reported serialization failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.