rstudio / rstudio/shinytest2

Snapshot of `file` to '...' has changed

Open
#281 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
123
Forks
23
PR merge metrics
No merged PRs in 30d

Description

I created a simple snapshot test:

library(shinytest2)

test_that("{shinytest2} recording: startup", {
  app <- AppDriver$new(variant = platform_variant(), name = "startup", seed = 3812, 
      height = 976, width = 1619)
  app$expect_screenshot()
})

...which passes when I run shinytest2::test_app() (mac-OS 12.6). However, my GitHub action on macos-latest fails:

✔ | F W S  OK | Context
    date, intersect, setdiff, union


⠏ |         0 | shinytest2                                                      
⠋ | 1       0 | shinytest2                                                      
✖ | 1       0 | shinytest2 [11.0s]
────────────────────────────────────────────────────────────────────────────────
Failure (test-shinytest2.R:7): {shinytest2} recording: startup
Snapshot of `file` to 'shinytest2/startup-001.png' has changed
Error: Error: Test failures
Run `testthat::snapshot_review('shinytest2/')` to review changes
Execution halted
Backtrace:
 1. app$expect_screenshot()
      at test-shinytest2.R:7:2
 2. shinytest2:::app_expect_screenshot_and_variant(...)
 3. shinytest2:::app_expect_screenshot(self, private, ...)
 4. shinytest2:::app__expect_snapshot_file(...)
 6. testthat::expect_snapshot_file(...)

My check-app.yaml file:

on:
  push:
    branches: [main, master]
  pull_request:
    branches: [main, master]

name: my_shiny_app

jobs:
  test-app:
    runs-on: ${{ matrix.os }}
    name: ${{ matrix.os }} (${{ matrix.r }})
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest]
        r: [4.2.1]
    env:
      GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
      R_KEEP_PKG_SOURCE: yes
    steps:
      - uses: actions/checkout@v2
      - uses: r-lib/actions/setup-pandoc@v2
      - uses: r-lib/actions/setup-r@v2
        with:
          r-version: ${{ matrix.r }}
          http-user-agent: ${{ matrix.config.http-user-agent }}
          use-public-rspm: true
      - uses: r-lib/actions/setup-renv@v2
      - uses: rstudio/shinytest2/actions/test-app@v1
        with:
          path: "."

The CI test OS: macOS 12.6.1

If I had to guess on what the problem is, I would bet on the png width slightly changing. It appears that the png width doesn't stay constant in all conditions, even though width= is set via AppDriver$new()

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The reproduction is in test-shinytest2.R at line 7, using AppDriver$new() and app$expect_screenshot(). Run shinytest2::test_app() locally and the rstudio/shinytest2/actions/test-app@v1 workflow on macos-latest, then compare shinytest2/startup-001.png and use testthat::snapshot_review('shinytest2/'). Done means identifying the cause of the platform-specific snapshot difference and establishing a reliable test result.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, r
Domain
ci-cd, operating-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.