high false positive rate in static site?
- Dominant language
- JavaScript
- Stars
- 7.2k
- Forks
- 621
- PR merge metrics
- No merged PRs in 30d
Description
Hello! Thanks for developing this tool. I'm trying to use it to automate visual regression tests for my static site, since it has lots of pages and I can't always keep an eye on everything. However, I'm unable to get anything working. I've set up backstop as per the documentation in the README, cross-checked with a "static site" example, and made adjustments that I thought would help -- all to no avail: __I am consistently getting false positives__.
Specifically, it seems like about 25% of all my pages do not load properly when either making references or comparing. One clear example of this is a page with default styling (evidently missing CSS), but there are other, more subtle problems (e.g., a page in which all text is italicised, for some reason?). There seems to be a fixed probability of this happening for any given page. As a result, after 15 runs, I haven't once had a clean report while creating a reference of a page, then comparing that same page, unchanged, to the reference.
I'm running backstop on a `python -m http.server` local server over HTTP. I have configured the load wait time to be 5 seconds (to give the chromium instance time to load all the assets etc). My site is less than 200kb total per page, so I can't imagine why it would even need that long. When I manually open the site in Chrome, and spam "Refresh without cache", there is visually no load time at all.
What am I doing wrong? Are the steps I can take to mitigate the weird loading failures? Is there something else I'm missing?
Thank you!
Example of a failure:
Contributor guide
Assessment
This issue has not been assessed yet.