lightly-ai / lightly-ai/lightly-train

[QUESTION] EoMT validation OOM because masks are resized to original image size

Open
#936 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Python
Stars
1.7k
Forks
116
Avg merge
2d 21h
Merged PRs (30d)
6

Description

Hi,

I am training `dinov3/vitt16plus-eomt-inst-coco` with LightlyTrain 0.16.4
for instance segmentation.

My original images are approximately 2943×1435, but I use:

```
```python
transform_args={
"image_size": (640, 1280),
"val": {
"image_size": (640, 1280),
},
}
```

I use batch_size=1 on an idle RTX 3060 with 12 GB VRAM. Training works
and uses approximately 2.5 GB, but validation fails with:

`torch.OutOfMemoryError: Tried to allocate 3.15 GiB`

The error occurs in get_labels_masks_scores() when the predicted masks
appear to be resized back to the original image resolution.
Is this intended behavior? Is there a supported way to calculate validation
metrics at the configured (640, 1280) resolution instead of the original
image size, without physically resizing the dataset?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at get_labels_masks_scores() in the validation path and reproduce the reported OOM with LightlyTrain 0.16.4, batch size 1, and the shown image-size settings. Trace where predicted masks are resized and determine whether validation can use the configured resolution. Done means the supported behavior is established and validation metrics run without the reported allocation failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.