lightly-ai / lightly-ai/lightly-train
[QUESTION] EoMT validation OOM because masks are resized to original image size
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 116
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 6
Description
Hi,
I am training `dinov3/vitt16plus-eomt-inst-coco` with LightlyTrain 0.16.4
for instance segmentation.
My original images are approximately 2943×1435, but I use:
```
```python
transform_args={
"image_size": (640, 1280),
"val": {
"image_size": (640, 1280),
},
}
```
I use batch_size=1 on an idle RTX 3060 with 12 GB VRAM. Training works
and uses approximately 2.5 GB, but validation fails with:
`torch.OutOfMemoryError: Tried to allocate 3.15 GiB`
The error occurs in get_labels_masks_scores() when the predicted masks
appear to be resized back to the original image resolution.
Is this intended behavior? Is there a supported way to calculate validation
metrics at the configured (640, 1280) resolution instead of the original
image size, without physically resizing the dataset?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at get_labels_masks_scores() in the validation path and reproduce the reported OOM with LightlyTrain 0.16.4, batch size 1, and the shown image-size settings. Trace where predicted masks are resized and determine whether validation can use the configured resolution. Done means the supported behavior is established and validation metrics run without the reported allocation failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100