Use pytest-xdist on CI

For local development, I am using [pytest-xdist](https://pypi.org/project/pytest-xdist/) very frequently to run our tests which distributes test execution on multiple subprocesses.

Some of our tests are not really robust to this and fail quite reliably (most offenders are due to fixed ports or due to process memory inspection).

I would love for our test suite to be able to run tests in parallel and be robust about it. I am not 100% sure if the effort is worth it for local development only (I started writing scripts to ignore these tests when running locally). However, I can see huge benefits for our CI runtime as well.
https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources

states that linux and windows runners typically have 2 CPUs while OSX runners have 3 CPUs. Particularly with us being bottlenecked on OSX builds, this could reduce CI runtimes significantly.

Thoughts? Concerns?