Visual-Regression-Tracker / Visual-Regression-Tracker/backend
VLM analysis does not respect ignore regions - baseline image sent without ignore area masking
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 15
- Forks
- 18
- Avg merge
- 15h 50m
- Merged PRs (30d)
- 2
Description
Problem
When using VLM analysis (e.g. Google Gemini), ignore regions defined in the
UI are not respected. This is because the original unmasked baseline image is
sent to the VLM, rather than a version with ignore regions applied.
As a result, the VLM flags differences in areas that have been explicitly
marked as ignore regions, leading to false positives.
Expected Behavior
When sending images to the VLM for analysis, the baseline image should have
ignore regions masked/blacked out before being sent. This way the VLM will
not analyze or flag differences in those areas.
Suggested Fix
Before sending the baseline image to the VLM:
- Retrieve the ignore regions for the corresponding TestVariation
- Apply the ignore region masks to the baseline image (e.g. fill with black
or grey overlay) - Send the masked baseline image to the VLM instead of the original
Current Behavior
- Ignore regions are applied at pixel diff level only
- VLM receives the original unmasked baseline image
- VLM flags changes in ignored areas as regressions
Environment
- VRT API version: 5.1.1
- VRT UI version: 5.1.2
- VLM: Google Gemini
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the VLM analysis path that sends the baseline image, then inspect how ignore regions are retrieved for the corresponding TestVariation. Verify that the image sent to the VLM has those regions masked and confirm that ignored differences no longer produce VLM regressions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 56/100