Visual-Regression-Tracker / Visual-Regression-Tracker/backend

VLM analysis does not respect ignore regions - baseline image sent without ignore area masking

Open
#348 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
15
Forks
18
Avg merge
15h 50m
Merged PRs (30d)
2

Description

Problem

When using VLM analysis (e.g. Google Gemini), ignore regions defined in the
UI are not respected. This is because the original unmasked baseline image is
sent to the VLM, rather than a version with ignore regions applied.

As a result, the VLM flags differences in areas that have been explicitly
marked as ignore regions, leading to false positives.

Expected Behavior

When sending images to the VLM for analysis, the baseline image should have
ignore regions masked/blacked out before being sent. This way the VLM will
not analyze or flag differences in those areas.

Suggested Fix

Before sending the baseline image to the VLM:

  1. Retrieve the ignore regions for the corresponding TestVariation
  2. Apply the ignore region masks to the baseline image (e.g. fill with black
    or grey overlay)
  3. Send the masked baseline image to the VLM instead of the original

Current Behavior

  • Ignore regions are applied at pixel diff level only
  • VLM receives the original unmasked baseline image
  • VLM flags changes in ignored areas as regressions

Environment

  • VRT API version: 5.1.1
  • VRT UI version: 5.1.2
  • VLM: Google Gemini

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the VLM analysis path that sends the baseline image, then inspect how ignore regions are retrieved for the corresponding TestVariation. Verify that the image sent to the VLM has those regions masked and confirm that ignored differences no longer produce VLM regressions.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
56/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.