garris / garris/BackstopJS

BackstopJS Not Detecting Missing UI Elements Despite Zero Mismatch Threshold - False Positive Issue

Open
#1,607 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
7.2k
Forks
621
PR merge metrics
No merged PRs in 30d

Description

# BackstopJS Not Detecting Missing UI Elements Despite Zero Mismatch Threshold - False Positive Issue

## Issue Description
BackstopJS is producing false positives by failing to detect visual regressions when UI elements are completely missing from the test image compared to the reference image, even with `misMatchThreshold` set to 0.

## Expected Behavior
When `misMatchThreshold` is set to 0, BackstopJS should detect any visual differences between the reference and test images, including missing UI elements. A completely missing "Send feedback" button should result in a test failure.

## Actual Behavior
BackstopJS reports the test as **PASSING** (false positive) when a significant UI element is completely missing from the test image. The "Send feedback" button is entirely absent in the test image but present in the reference image, yet the comparison passes.

## Configuration Tested
```json
{
"misMatchThreshold": 0,
"requireSameDimensions": false
}
```

**Also tested with:**
```json
{
"misMatchThreshold": 0,
"requireSameDimensions": true
}
```

## Steps to Reproduce
1. Set up BackstopJS configuration with `misMatchThreshold: 0` and `requireSameDimensions: false`
2. Create a reference image with complete UI including the "Send feedback" button
3. Generate a test scenario where the "Send feedback" button is missing/removed
4. Run BackstopJS visual comparison
5. **Result:** Test passes despite obvious missing button

## Environment
- BackstopJS Version: [6.3.25] Also tried out with `6.2.2`
- Browser: [Chromium]
- OS: [MAC OS]

## Detailed Findings

### With `requireSameDimensions: false` (Current Issue)
- Test **passes** despite missing "Send feedback" button
- No visual regression detected
- Images have different heights due to missing button
- This is the main issue - **false positive**

### With `requireSameDimensions: true` (Also Problematic)
- Test **fails** due to dimension mismatch
- BUT fails for wrong reason (dimension difference, not missing button)
- Doesn't specifically identify what visual content is different
- Catches the height change but not the semantic difference

## Screenshots

Image

## Core Problem
BackstopJS appears to have issues with:
1. **Pixel-by-pixel comparison** when images have different dimensions
2. **Missing content detection** at page boundaries

## Questions for Maintainers
1. How does BackstopJS handle pixel comparison when images have different heights?
2. Is this expected behavior when `requireSameDimensions: false`?

## Expected Fix/Enhancement
BackstopJS should detect missing UI elements as visual regressions regardless of dimension differences when `misMatchThreshold` is set to 0. The comparison algorithm should account for missing buttons and other UI elements, flagging them appropriately.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.