Feature Request: Parallel Test Execution with pytest-split

Summary

Implement parallel test execution in CI using pytest-split to reduce overall test run time by distributing tests across multiple concurrent jobs.

Motivation

As test suites grow, CI run times increase proportionally. Running all tests sequentially can lead to:

• Long feedback loops for developers
• Increased CI costs (longer runner time)
• Slower deployment velocity

By splitting tests across multiple parallel jobs, we can significantly reduce wall-clock time while maintaining full test coverage.

Implementation Overview

The solution uses three key components:

1. pytest-split plugin - Splits tests into groups based on historical duration data
2. GitHub Actions matrix strategy - Runs multiple test groups in parallel
3. Duration tracking workflow - Keeps test timing data up-to-date for optimal splitting

---

Implementation Details

1. Install pytest-split

Add pytest-split to your test dependencies.

If using pip/requirements:

pytest-split

If using pixi (conda-forge):

[feature.test.dependencies]
pytest-split = "*"

2. Create the CI Workflow

Create .github/workflows/test.yml:

name: Unit tests

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

concurrency:
  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: true

jobs:
  pytest:
    name: Run pytest
    runs-on: ubuntu-latest
    timeout-minutes: 60
    strategy:
      fail-fast: false
      matrix:
        group: [1, 2, 3, 4, 5, 6, 7, 8]  # Adjust number of groups as needed

    steps:
    - name: Checkout
      uses: actions/checkout@v4

    - name: Set up Python
      uses: actions/setup-python@v5
      with:
        python-version: '3.11'

    - name: Install dependencies
      run: pip install -r requirements.txt

    - name: Run tests
      run: |
        pytest -v --splits ${{ strategy.job-total }} --group ${{ matrix.group }}

  all_tests_pass:
    if: always()
    runs-on: ubuntu-latest
    name: All tests pass
    needs: [pytest]
    steps:
    - name: Check build matrix status
      if: needs.pytest.result != 'success'
      run: exit 1

Key Configuration Options

Option	Description
--splits N	Total number of groups to split into (use ${{ strategy.job-total }} for automatic)
--group M	Which group this job should run (1-indexed)
--durations-path FILE	Path to durations file (defaults to .test_durations)
fail-fast: false	Continue running other groups even if one fails

3. Create Duration Update Workflow (Separate Action)

The splitting algorithm works best when it knows how long each test takes. Without duration data, pytest-split falls back to naive splitting by test count, which can result in unbalanced groups (e.g., one group gets all the slow integration tests).

This is implemented as a separate GitHub Action workflow that:

1. Runs on a schedule (weekly) to capture updated test timings
2. Executes the full test suite with --store-durations to record how long each test takes
3. Automatically creates a PR with the updated .test_durations file
4. Allows human review before merging the timing updates

Why a Separate Workflow?

Reason	Explanation
Avoids CI bloat	Recording durations adds overhead; don't do it on every PR
Clean separation	Test execution (PR workflow) vs. maintenance (duration updates)
Human review	Duration changes go through PR review, not auto-committed
Scheduled updates	Weekly cadence catches timing drift as tests evolve
Manual trigger	Can run on-demand after major test suite changes

Workflow: .github/workflows/update-test-durations.yml

name: Update Test Durations

on:
  schedule:
    - cron: '0 9 * * 1'  # Weekly on Monday at 9:00 AM UTC
  workflow_dispatch:      # Allow manual trigger

permissions:
  contents: write         # Needed to create commits
  pull-requests: write    # Needed to create PRs

jobs:
  update-test-durations:
    runs-on: ubuntu-latest

    steps:
    # Step 1: Check out the repository
    - name: Checkout repository
      uses: actions/checkout@v4
      with:
        fetch-depth: 0    # Full history for accurate branch creation

    # Step 2: Set up your environment (adjust for your setup)
    - name: Set up Python
      uses: actions/setup-python@v5
      with:
        python-version: '3.11'

    - name: Install dependencies
      run: pip install -r requirements.txt

    # Step 3: Run the full test suite with --store-durations
    # This writes/updates the .test_durations file with timing data
    - name: Run tests to update durations
      run: pytest --store-durations -v
      continue-on-error: true  # Continue even if some tests fail
                               # We still want partial duration data

    # Step 4 (Optional): Fix formatting issues with the durations file
    # The JSON file may need trailing newline or formatting fixes
    - name: Fix and verify formatting
      run: |
        # Run pre-commit to auto-fix any formatting issues
        pre-commit run --files .test_durations || true
        # Run again to verify the file now passes all checks
        pre-commit run --files .test_durations

    # Step 5: Create a Pull Request with the updated durations
    - name: Create Pull Request
      uses: peter-evans/create-pull-request@v6
      with:
        token: ${{ secrets.GITHUB_TOKEN }}
        commit-message: 'chore: Update test duration timings'
        title: 'chore: Update test duration timings'
        body: |
          This PR updates the test duration timings stored in `.test_durations`.
          
          The test durations are used by `pytest-split` to distribute tests 
          across parallel CI jobs for optimal performance.
          
          This is an automated update generated by the weekly test durations workflow.
          
          Please review and merge if the changes look reasonable.
        branch: automated/update-test-durations
        delete-branch: true   # Clean up branch after merge
        labels: |
          automated
          chore

Key Flags Explained

Flag/Setting	Purpose
--store-durations	pytest-split flag that writes test timings to .test_durations
continue-on-error: true	Capture timing data even if some tests fail
add-paths: .test_durations	Only include the durations file in the PR (not other changes)
delete-branch: true	Auto-cleanup the PR branch after merge
workflow_dispatch	Allows manual trigger from GitHub Actions UI

4. Initialize the Durations File

Generate the initial .test_durations file locally:

pytest --store-durations -v

This creates a JSON file mapping each test to its execution time:

{
    "tests/test_example.py::test_fast": 0.001,
    "tests/test_example.py::test_slow": 5.234,
    ...
}

Commit this file to your repository.

---

Optional: Coverage Aggregation

If using coverage, each parallel job produces a partial coverage report. Aggregate them:

jobs:
  pytest:
    # ... (as above)
    steps:
    - name: Run tests with coverage
      run: |
        coverage run --source=src -m pytest -v --splits ${{ strategy.job-total }} --group ${{ matrix.group }}
      env:
        COVERAGE_FILE: cov-report-${{ matrix.group }}

    - name: Upload coverage data
      uses: actions/upload-artifact@v4
      with:
        name: cov-report-${{ matrix.group }}
        path: cov-report-${{ matrix.group }}
        retention-days: 1

  combine-coverage:
    name: Combine coverage reports
    needs: pytest
    runs-on: ubuntu-latest

    steps:
    - name: Checkout
      uses: actions/checkout@v4

    - name: Download all coverage data
      uses: actions/download-artifact@v4
      with:
        pattern: cov-report-*
        merge-multiple: true

    - name: Combine and report
      run: |
        pip install coverage
        coverage combine cov-report-*
        coverage xml -o coverage.xml
        coverage report -m

---

Pros and Cons

✅ Pros

Benefit	Description
Faster CI	Wall-clock time reduced by ~N× with N parallel jobs
Optimal splitting	Duration-aware algorithm balances work across groups
Automatic rebalancing	Weekly updates adjust for test time changes
No test changes needed	Works with existing pytest tests, no code modifications
Fail-fast optional	Can continue other groups or fail immediately
Scales easily	Just change the matrix group count to add/remove parallelism
Coverage compatible	Works with coverage by combining partial reports

❌ Cons

Drawback	Description
Increased runner usage	N parallel jobs consume N× compute minutes
Maintenance overhead	Duration file needs periodic updates
Merge conflicts	.test_durations can have conflicts when tests change
Cold start	First run without duration data uses naive splitting
Artifact management	Coverage requires upload/download of partial reports
Test isolation required	Tests must be independent; shared state can cause issues
Debugging complexity	Failures split across jobs can be harder to investigate

---

Recommendations

1. Start with 4-8 groups - More groups have diminishing returns and increase overhead
2. Monitor runner costs - Parallel jobs use concurrent minutes; ensure your plan supports it
3. Keep tests independent - Ensure no test depends on another's side effects
4. Review duration PRs - Automated duration updates should be reviewed before merge
5. Use fail-fast: false - See all failures at once rather than stopping at first

---

References

• pytest-split documentation
• GitHub Actions matrix strategy
• peter-evans/create-pull-request action

Start by inspecting the repository's existing dependency configuration and GitHub Actions workflows before adding the proposed .github/workflows/test.yml and .github/workflows/update-test-durations.yml. Check how the project runs pytest and whether requirements.txt or pixi configuration is used. Done means CI runs all test groups, duration data is updated through the scheduled workflow, and the resulting status is reported correctly.