lightly-ai / lightly-ai/lightly

VICRegL error when grid size is too big

Open
#1,179 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
3.8k
Forks
367
Avg merge
3d 22h
Merged PRs (30d)
5

Description

## Reproduce
Run https://docs.lightly.ai/self-supervised-learning/examples/vicregl.html with `transform = VICRegLTransform(n_local_views=0, global_grid_size=9)` (default is `global_grid_size=7`).

If you run it on GPU, you'll get a very ugly CUDA error that doesn't say much. If you try it on CPU, you can backtrace it.

## Description
Calculating [`nearest_neighbors`](https://github.com/lightly-ai/lightly/blob/dda9e8405d1460271e18d5200330b94bfb87c39f/lightly/models/utils.py#L474) in `VICRegLLoss`'s [`_nearest_neighbors_on_l2`](https://github.com/lightly-ai/lightly/blob/dda9e8405d1460271e18d5200330b94bfb87c39f/lightly/loss/vicregl_loss.py#L387) and [`_nearest_neighbors_on_grid`](https://github.com/lightly-ai/lightly/blob/dda9e8405d1460271e18d5200330b94bfb87c39f/lightly/loss/vicregl_loss.py#L420) causes the above error. What happens is that the calculated indices in `min_indices` are *sometimes* out of bound for the given input: https://github.com/lightly-ai/lightly/blob/dda9e8405d1460271e18d5200330b94bfb87c39f/lightly/models/utils.py#L518

That's as far as I've gotten, I haven't looked further into it and do not understand what exactly is going on. I believe that your implementation is correct, but it would be useful to catch the error when the specified grid size is too big and provide a useful error message to the user.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the VICRegL example with VICRegLTransform(n_local_views=0, global_grid_size=9), then inspect lightly/loss/vicregl_loss.py in _nearest_neighbors_on_l2 and _nearest_neighbors_on_grid. Trace the nearest_neighbors call in lightly/models/utils.py, especially the min_indices use. Done means an oversized grid produces a clear user-facing error instead of the CUDA or index-out-of-bounds failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.