synthesizer跑到到1500步就出错，有高手知道是怎么回事吗

下面是日志：
{| Epoch: 143/910 (10/11) | Loss: 0.3434 | 1.1 steps/s | Step: 1k | }{| Epoch: 143/910 (11/11) | Loss: 0.3442 | 1.1 steps/s | Step: 1k | }
{| Epoch: 144/910 (11/11) | Loss: 0.3435 | 1.1 steps/s | Step: 1k | }
{| Epoch: 145/910 (2/11) | Loss: 0.3433 | 1.1 steps/s | Step: 1k | }Traceback (most recent call last):
File "D:\MockingBird-main\synthesizer_train.py", line 37, in
train(**vars(args))
File "D:\MockingBird-main\synthesizer\train.py", line 201, in train
loss.backward()
File "D:\anaconda\envs\pytorch\lib\site-packages\torch\_tensor.py", line 307, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "D:\anaconda\envs\pytorch\lib\site-packages\torch\autograd\__init__.py", line 154, in backward
Variable._execution_engine.run_backward(
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

Start with synthesizer_train.py and synthesizer/train.py, especially the loss.backward() call at line 201, and review the reported traceback and training-step handling. Reproduce the failure around the reported step with the same training setup; done means identifying a reproducible cause and documenting a verified resolution or limitation.

Written by the indexing model from the issue text.