GoogleChrome / GoogleChrome/lighthouse

Lighthouse fails to produce accessibility score altogether for incomplete axe-core result

Open
#15,828 4 comments 1 reaction 1 assignee Claimed by @paulirish View on GitHub
bug P1.5 upstream-action-required
Dominant language
JavaScript
Stars
30.8k
Forks
9.8k
Avg merge
1d 14h
Merged PRs (30d)
20

Description

### FAQ

- [X] Yes, my issue is not about [variability](https://github.com/GoogleChrome/lighthouse/blob/main/docs/variability.md) or [throttling](https://github.com/GoogleChrome/lighthouse/blob/main/docs/throttling.md).
- [X] Yes, my issue is not about a specific accessibility audit (file with [axe-core](https://github.com/dequelabs/axe-core) instead).

### URL

https://axecore-lighthouse-failure.glitch.me/

### What happened?

Running lighthouse on the above page fails to produce any accessibility score at all:

![image](https://github.com/GoogleChrome/lighthouse/assets/1588923/03ae9ee6-ac23-4885-9cb8-b9b25d359cf2)

### What did you expect?

It would be preferable for lighthouse to be robust to errors in axe-core and still produce an accessibility score.

We try to gate code submissions on accessibility scores and while lighthouse / axe-core reports plenty of other successful accessibility checks, it is completely masked by the incomplete check (in-terms of gating on accessibility checks and tracking history of the score).

I would expect:
- An accessibility score to still be calculated to prevent breaking PR gates completely based on score and fail to see general progress
- A link / more resources to report the incomplete result to axe-core

### What have you tried?

Isolated and disabled lighthouse testing for pages using the component that causes the lighthouse failure.
Reported the incomplete result and reproducing example to axe-core: https://github.com/dequelabs/axe-core/issues/4335
But would still like lighthouse to be more robust to axe-core errors and generate a useful score (might be a bit lower because of the error or maybe the error does not contribute to the score).

### How were you running Lighthouse?

PageSpeed Insights, Chrome DevTools

### Lighthouse Version

11.3.0

### Chrome Version

121.0.6167.185

### Node Version

_No response_

### OS

_No response_

### Relevant log output

```shell
axe-core Error: Cannot read properties of undefined (reading 'props')
```

as mentioned above, the axe-core error was reported at: https://github.com/dequelabs/axe-core/issues/4335
but this issue is for improving lighthouses score calculation robustness to axe-core errors

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.