Tests are incorrectly split when there are a lot of slow skipped tests

I'm running `pytest-split` on a test suite that has a lot of slow tests which are `pytest.mark`ed as slow and skipped by default. I can include all these tests by using a custom `pytest --all-tests` option.
The issue I'm seeing now is that when I create a `.test_durations` file via `pytest --all-tests --store-durations` then the regular `pytest --splits 4 ...` run without the `--all-tests` will be unbalanced. It seems as though `pytest-split` is taking into account the runtime of the skipped tests and thus failing to produce a balanced split. A workaround is to store one `.tests_durations` file for `pytest --all-tests` and a separate file for the plain `pytest`, but I think ideally this should not be required.