`shinytest2` approach to saving/comparing images

This issue is to propose a new approach to how images are store/compared in an attempt to dramatically reduce the number of snapshot file, reducing overall confusion.

• Current approach:

  • Store all snaps in _snaps/OS-RVersion/TEST/XYZ.{json,png}
  • Checking:
    • Given an OS/RVersion, the json and images much match perfectly
  • Pros:
    • Get json/images for each and every OS/RVersion combo
    • Can have different, known output for different OS/RVersions
  • Cons:
    • More files to maintain. Becomes unwieldily very quickly with many tests
    • Hard to see differences between images within the same test

• New approach:

  • Store all snaps in _snaps/TEST/XYZ.{json,png} (No variant)
  • Store the master version meta information in _shinytest2.json. Info such as OS and RVersion.
  • Checking:
    • If the OS/RVersion match exactly, then all images / json should match perfectly
    • Else, all images/json should be able to fuzzy match
      • Image fuzzy matching should be similar to https://github.com/MangoTheCat/visualTest . Inspiration: https://github.com/rstudio/shinytest/issues/412
      • JSON:
        • Maybe the HTML SHA should never be compared?
        • Maybe the image SHA should never be compared?
        • Maybe we fuzzy match the information?
  • Pros:
    • Minimal number of files to maintain; 1:1 ratio of json/images to expected snapshots
    • Do not let snapshots from other platforms slip through the cracks as long as the master version exists.
      Example using the "current approach" above: Init on mac and then test on linux. This situation will never throw on differences
      However, using fuzzy matching, you can init on mac and then immediately test on linux using fuzzy matching.
      Shinycoreci-apps ran into this due to maintenance fatigue and accepting images right away. Then not being able to compare json/images between OS/RVersions allowed the image to survive longer than it should have.
  • Cons:
    • Exact images for all flavors are not known. Only the master version
    • Small display issues in other OS/RVersions will not be caught. Ex: Let's say linux displays sliders using dark grey, while the master version displays them with light grey. The fuzzy match will say that it is tolerable, while it might actually be an issue.
      If this is the case, the tolerance could be set to 0, but would have be manually done.

• st2_expect_html():

  • Testing functions like st2_expect_html() would also be bundled into this situation.
  • I would argue that st2_expect_html() should not have fuzzy matching by default. Maybe it should be run through tagQuery() (or something similar) to produce consistent output. Ex: attribs$class names and attribs keys are in a consistent order.

@wch What are your thoughts?

I believe this is a win/win in that we get less images to maintain and we also get to fuzzy match when testing on differing OS/RVersions. (Exact matching on the master OS/RVersion; Fuzzy matching on all other OS/RVersions.) This would also allow for Linux machines to be bold and not bold and not provide false-positive results in shinycoreci-apps.

cc @MadhulikaTanuboddi

Start by reviewing the current _snaps/OS-RVersion/TEST layout and the proposed _snaps/TEST layout, including the _shinytest2.json metadata. Examine how st2_expect_html() is expected to fit into the approach, then resolve the exact-versus-fuzzy matching rules for images and JSON. Done means the storage format and matching behavior are defined clearly enough to implement and test.