lightly-ai / lightly-ai/lightly
torch._C._LinAlgError: linalg.svd: (Batch element 0): The algorithm failed to converge because the input matrix is ill-conditioned or has too many repeated singular values (error code: 15).
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.8k
- Forks
- 367
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 5
Description
C:\Users\LocalAdmin\anaconda3\envs\lightlyyolo\python.exe D:\Charis\SSL-yolo8\lightly-master\examples\pytorch\mmcr_yolo.py
WARNING ⚠️ no model scale passed. Assuming scale='n'.
class_name is: MMCR
save_path is: D:\Charis\SSL-yolo8\lightly-master\runs\MMCR
Starting Training
epoch: 00, loss: -2415920191337764664519950336.00000
after training
tensor([ -0.7926, -2.2815, -0.7858, -14.8213, -16.7507], device='cuda:0')
tensor([ -0.7926, -2.2815, -0.7858, -14.8213, -16.7507], device='cuda:0')
tensor([ -0.7926, -2.2815, -0.7858, -14.8213, -16.7507], device='cuda:0')
tensor([-0.4687, -0.7416, -0.3247, -4.7035, -5.2732], device='cuda:0')
after saving training + has backbone.load_state_dict
tensor([-0.4687, -0.7416, -0.3247, -4.7035, -5.2732], device='cuda:0')
tensor([-0.4687, -0.7416, -0.3247, -4.7035, -5.2732], device='cuda:0')
tensor([-0.4687, -0.7416, -0.3247, -4.7035, -5.2732], device='cuda:0')
tensor([-0.4687, -0.7416, -0.3247, -4.7035, -5.2732], device='cuda:0')
save full_path is: D:\Charis\SSL-yolo8\lightly-master\runs\MMCR\MMCR_coca_alldcm_MMCRTransform.pth
Saving model for MMCR_coca_alldcm_MMCRTransform.pth at Epoch 1
Finding optimal model params. Loss is dropping from -2415920191337764664519950336.0000 to -2415920191337764664519950336.0000
D:\Charis\SSL-yolo8\lightly-master\lightly\loss\mmcr_loss.py:60: UserWarning: torch.linalg.svd: During SVD computation with the selected cusolver driver, batches 0, 1, 2, 3, 4, and other 123 batches failed to converge. A more accurate method will be used to compute the SVD as a fallback. Check doc at https://pytorch.org/docs/stable/generated/torch.linalg.svd.html (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\linalg\BatchLinearAlgebraLib.cpp:703.)
_, S_z, _ = svd(z)
Traceback (most recent call last):
File "D:\Charis\SSL-yolo8\lightly-master\examples\pytorch\mmcr_yolo.py", line 158, in
loss = criterion(z_o, z_m)
File "C:\Users\LocalAdmin\anaconda3\envs\lightlyyolo\lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\LocalAdmin\anaconda3\envs\lightlyyolo\lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "D:\Charis\SSL-yolo8\lightly-master\lightly\loss\mmcr_loss.py", line 60, in forward
_, S_z, _ = svd(z)
torch._C._LinAlgError: linalg.svd: (Batch element 0): The algorithm failed to converge because the input matrix is ill-conditioned or has too many repeated singular values (error code: 15).
Process finished with exit code 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure from examples/pytorch/mmcr_yolo.py and inspect lightly/loss/mmcr_loss.py at line 60, where the SVD error occurs. First examine the extreme loss value and the tensors passed to the loss; done means identifying the cause and confirming that training completes without the convergence error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100