Tests are incorrectly split when there are a lot of slow skipped tests

I'm running `pytest-split` on a test suite that has a lot of slow tests which are `pytest.mark`ed as slow and skipped by default. I can include all these tests by using a custom `pytest --all-tests` option.
The issue I'm seeing now is that when I create a `.test_durations` file via `pytest --all-tests --store-durations` then the regular `pytest --splits 4 ...` run without the `--all-tests` will be unbalanced. It seems as though `pytest-split` is taking into account the runtime of the skipped tests and thus failing to produce a balanced split. A workaround is to store one `.tests_durations` file for `pytest --all-tests` and a separate file for the plain `pytest`, but I think ideally this should not be required.

Reproduce the imbalance with pytest --all-tests --store-durations followed by a regular pytest --splits 4 run, then trace how stored durations for skipped tests are loaded and used for splitting. Done means the regular run produces balanced splits without requiring a separate duration file, with coverage for suites containing slow tests skipped by default.

索引モデルが issue の本文から書いたものです。