Two self-hosted runners incorrectly run the same job

Describe the bug
If you launch a multiple of workflows that run on selfhosted runners, sometimes, two runners will run the same job. What this means is that if we have self hosted Runner A and Runner B and Workflow A and Workflow B, Runner A and Runner B will both run Workflow A leaving no runners to run Workflow B.

The GHA UI only shows the output from one of the runners, if looking at the logs on the other runner, we can see it is also running Workflow A and Workflow B remains queued.

To Reproduce
This doesn't happen all the time, but it definitely happens at least once every ~50 attempts
Steps to reproduce the behavior:

1. Launch multiple workflows that run on self hosted runners
2. Watch two self hosted runners pick up and run the same workflow.

Expected behavior
Only one runner picks up one workflow.

Runner Version and Platform

Version of your runner?
v2.311.0

OS of the machine running the runner? OSX/Windows/Linux/...
Linux x86_64

Start by reproducing the race with multiple workflows on self-hosted runners, using runner version v2.311.0 on Linux x86_64. Compare the GitHub Actions UI with logs from both runners when the duplication occurs. Done means each workflow is picked up by only one runner and other queued workflows remain available for another runner.