chartjs / chartjs/Chart.js

Image-Based Unit Tests Fail on Mac

Open
#11,828 0 comments 1 reaction 0 assignees View on GitHub
type: bug
Dominant language
JavaScript
Stars
67.7k
Forks
11.9k
Avg merge
7h 39m
Merged PRs (30d)
5

Description

### Expected behavior

Unit tests should be green.

### Current behavior

Pulling the latest master and run `pnpm test` I get this:

```
[karma] Chrome 126.0.0.0 (Mac OS 10.15.7) Time scale tests auto /base/test/fixtures/scale.timeseries/data-timestamps.js FAILED
[karma] Fixture test failed:
[karma] Difference: 1166px / 0.44%
[karma] Threshold: 1%
[karma] Tolerance: 0.15%
[karma] Error: Fixture test failed:
[karma] Difference: 1166px / 0.44%
[karma] Threshold: 1%
[karma] Tolerance: 0.15%
[karma] at
[karma] at node_modules/.pnpm/chartjs-test-utils@0.4.0_jasmine@3.99.0_karma-jasmine@4.0.2_karma@6.4.1/node_modules/chartjs-test-utils/dist/chartjs-test-utils.js:1086:25 <- test/index.js:1091:27
[karma] at image.onload (node_modules/.pnpm/chartjs-test-utils@0.4.0_jasmine@3.99.0_karma-jasmine@4.0.2_karma@6.4.1/node_modules/chartjs-test-utils/dist/chartjs-test-utils.js:527:5 <- test/index.js:532:7)
[karma] Chrome 126.0.0.0 (Mac OS 10.15.7) Time scale tests auto /base/test/fixtures/scale.timeseries/data-timestamps.js FAILED
[karma] Fixture test failed:
[karma] Difference: 1166px / 0.44%
[karma] Threshold: 1%
[karma] Tolerance: 0.15%
[karma] Error: Fixture test failed:
[karma] Difference: 1166px / 0.44%
[karma] Threshold: 1%
[karma] Tolerance: 0.15%
[karma] at
[karma] at node_modules/.pnpm/chartjs-test-utils@0.4.0_jasmine@3.99.0_karma-jasmine@4.0.2_karma@6.4.1/node_modules/chartjs-test-utils/dist/chartjs-test-utils.js:1086:25 <- test/index.js:1091:27
[karma] at image.onload (node_modules/.pnpm/chartjs-test-utils@0.4.0_jasmine@3.99.0_karma-jasmine@4.0.2_karma@6.4.1/node_modules/chartjs-test-utils/dist/chartjs-test-utils.js:527:5 <- test/index.js:532:7)
[karma] Chrome 126.0.0.0 (Mac OS 10.15.7): Randomized with seed 00514
[karma]
[karma] Firefox 115.0 (Mac OS 10.15): Executed 50 of 1649 (1 FAILED) (2.982 secs / 2.85 secs)
[karma] Chrome 126.0.0.0 (Mac OS 10.15.7): Executed 1009 of 1649 (1 FAILED) (9.784 secs / 8.73 secs)
[karma] TOTAL: 2 FAILED, 1057 SUCCESS
```

We also use image-based tests for our Chart.js use case and had similar issues.

The reason was, that each OS uses different fonts. So on Windows, Linux and Mac, the Chart.Js fonts for scale lables etc. look slightly different.

### Reproducible sample

---

### Optional extra steps/info to reproduce

_No response_

### Possible solution

We actually "solved" this by making Linux the default and starting the tests in a Linux docker container on other environments to get the same result.

Allowing an error threshold >0% for Chart.js is not a good idea as this causes false positives when e.g. a 1px line in a line chart has the wrong color.

### Context

_No response_

### chart.js version

v4.4.3

### Browser name and version

_No response_

### Link to your project

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.