Image-Based Unit Tests Fail on Mac
- Dominant language
- JavaScript
- Stars
- 67.7k
- Forks
- 11.9k
- Avg merge
- 7h 39m
- Merged PRs (30d)
- 5
Description
### Expected behavior
Unit tests should be green.
### Current behavior
Pulling the latest master and run `pnpm test` I get this:
```
[karma] Chrome 126.0.0.0 (Mac OS 10.15.7) Time scale tests auto /base/test/fixtures/scale.timeseries/data-timestamps.js FAILED
[karma] Fixture test failed:
[karma] Difference: 1166px / 0.44%
[karma] Threshold: 1%
[karma] Tolerance: 0.15%
[karma] Error: Fixture test failed:
[karma] Difference: 1166px / 0.44%
[karma] Threshold: 1%
[karma] Tolerance: 0.15%
[karma] at
[karma] at node_modules/.pnpm/chartjs-test-utils@0.4.0_jasmine@3.99.0_karma-jasmine@4.0.2_karma@6.4.1/node_modules/chartjs-test-utils/dist/chartjs-test-utils.js:1086:25 <- test/index.js:1091:27
[karma] at image.onload (node_modules/.pnpm/chartjs-test-utils@0.4.0_jasmine@3.99.0_karma-jasmine@4.0.2_karma@6.4.1/node_modules/chartjs-test-utils/dist/chartjs-test-utils.js:527:5 <- test/index.js:532:7)
[karma] Chrome 126.0.0.0 (Mac OS 10.15.7) Time scale tests auto /base/test/fixtures/scale.timeseries/data-timestamps.js FAILED
[karma] Fixture test failed:
[karma] Difference: 1166px / 0.44%
[karma] Threshold: 1%
[karma] Tolerance: 0.15%
[karma] Error: Fixture test failed:
[karma] Difference: 1166px / 0.44%
[karma] Threshold: 1%
[karma] Tolerance: 0.15%
[karma] at
[karma] at node_modules/.pnpm/chartjs-test-utils@0.4.0_jasmine@3.99.0_karma-jasmine@4.0.2_karma@6.4.1/node_modules/chartjs-test-utils/dist/chartjs-test-utils.js:1086:25 <- test/index.js:1091:27
[karma] at image.onload (node_modules/.pnpm/chartjs-test-utils@0.4.0_jasmine@3.99.0_karma-jasmine@4.0.2_karma@6.4.1/node_modules/chartjs-test-utils/dist/chartjs-test-utils.js:527:5 <- test/index.js:532:7)
[karma] Chrome 126.0.0.0 (Mac OS 10.15.7): Randomized with seed 00514
[karma]
[karma] Firefox 115.0 (Mac OS 10.15): Executed 50 of 1649 (1 FAILED) (2.982 secs / 2.85 secs)
[karma] Chrome 126.0.0.0 (Mac OS 10.15.7): Executed 1009 of 1649 (1 FAILED) (9.784 secs / 8.73 secs)
[karma] TOTAL: 2 FAILED, 1057 SUCCESS
```
We also use image-based tests for our Chart.js use case and had similar issues.
The reason was, that each OS uses different fonts. So on Windows, Linux and Mac, the Chart.Js fonts for scale lables etc. look slightly different.
### Reproducible sample
---
### Optional extra steps/info to reproduce
_No response_
### Possible solution
We actually "solved" this by making Linux the default and starting the tests in a Linux docker container on other environments to get the same result.
Allowing an error threshold >0% for Chart.js is not a good idea as this causes false positives when e.g. a 1px line in a line chart has the wrong color.
### Context
_No response_
### chart.js version
v4.4.3
### Browser name and version
_No response_
### Link to your project
_No response_
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.