feat: Improved test runner on large apps / projects

Description

[!NOTE]
As a developer of a large app project, I want to have confidence fast that my changes are working and then know exactly which changes have failed.

With Dart workspaces projects may look like the following:

app/
  pubspec.yaml
  test/
  packages/
    core/
      core_library_1/
        pubspec.yaml
        test/
      core_library_2/
        pubspec.yaml
        test/
    features/
      feature_a/
        pubspec.yaml → depends only on core_library_1 (via `path: ../../core/core_libary_1` import)
        test/
      feature_b/
        pubspec.yaml → depends only on core_library_2 (via `path: ../../core/core_libary_2` import)
        test/

A typical, simple way to run those test would be by simply calling:

very_good_cli:very_good test --recursive --coverage from within the app/ folder to run all tests.

There are several issues with this:

• It runs all tests, all the time.
• When it runs all tests, the test results are all appended one after the other. This can lead to weird situations where the check fails, but it says 00:51 +224: All tests passed! at the end, because the last test suite passed, while some earlier test suite failed.
• It runs all test one after another, instead of parallelising them on different machines.

Now imagine the following developer experience:

• You open a PR making changes only within the core_library_2 package.
• very_good_cli detects that changes were only done within core_library_2 and it detects that feature_b depends on core_library_2 → so it will test both core_library_2 and feature_b
• It spins out a test matrix, starting two GitHub Action checks for core_library_2 and feature_b
• Each package test run in isolation, get their own success/fail status and dedicated logs
• Each package reports their own coverage, coverage from untested packages are carried forward
• Once all packages checks are finished a check combining all test results is created with a failed status when at least one package failed

This is how an example GitHub Action workflow could look like:

name: CI

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

jobs:
  detect-affected:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.affected.outputs.matrix }}
      has_affected: ${{ steps.affected.outputs.has_affected }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0 # Requires full history to diff against target branch

      - uses: subosito/flutter-action@v2
        with:
          channel: 'stable'

      - name: Install Very Good CLI
        run: dart pub global activate very_good_cli

      - name: Detect Affected Workspace Packages
        id: affected
        run: |
          # Hypothetical CLI command: inspects Git diff against target branch,
          # maps workspace dependencies, and outputs JSON array of targets:
          # [{"name": "core_library_2", "path": "packages/core/core_library_2"}, ...]
          MATRIX_JSON=$(very_good workspace matrix --base origin/${{ github.base_ref || 'main' }})
          
          echo "matrix=$MATRIX_JSON" >> $GITHUB_OUTPUT
          
          if [ "$MATRIX_JSON" = "[]" ]; then
            echo "has_affected=false" >> $GITHUB_OUTPUT
          else
            echo "has_affected=true" >> $GITHUB_OUTPUT
          fi

  test:
    needs: detect-affected
    if: ${{ needs.detect-affected.outputs.has_affected == 'true' }}
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false # Allows all matrix jobs to complete so developers see all failures
      matrix:
        package: ${{ fromJson(needs.detect-affected.outputs.matrix) }}

    steps:
      - uses: actions/checkout@v4

      - uses: subosito/flutter-action@v2
        with:
          channel: 'stable'

      - name: Install Very Good CLI
        run: dart pub global activate very_good_cli

      - name: unit tests - ${{ matrix.package.name }}
        working-directory: ${{ matrix.package.path }}
        run: |
          very_good test --coverage

      - name: Upload Coverage to Codecov
        uses: codecov/codecov-action@v4
        working-directory: ${{ matrix.package.path }}
        with:
          files: ${{ matrix.package.path }}/coverage/lcov.info
          flags: ${{ matrix.package.name }} # Essential for Codecov Carryforward Flags
          token: ${{ secrets.CODECOV_TOKEN }}

  ci-status:
    name: unit tests
    needs: [detect-affected, test]
    if: always()
    runs-on: ubuntu-latest
    steps:
      - name: Evaluate Aggregate Matrix Status
        run: |
          if [ "${{ needs.test.result }}" == "failure" ] || [ "${{ needs.detect-affected.result }}" == "failure" ]; then
            echo "One or more package checks failed."
            exit 1
          fi
          echo "All affected package tests passed successfully!"

To enable this only one new command would have to be added:

very_good workspace matrix --base origin/main

returning the following information:

[{"name": "core_library_2", "path": "packages/core/core_library_2"}, ...]

Requirements

• All CI/CD checks are passing.
• There is no drop in the test coverage percentage.
• Adds a new command very_good workspace matrix.
• The command accepts a --base argument allowing it to diff the changes between the base ref and the current state.
• The command returns a JSON list of affected projects (all projects whose tests need to re-run).
• The returned JSON list contains the name (from the pubspec.yaml) and the path (relative to the current directory) of the project.
• The affected projects are determined based on whether changes have been done within the project compared to --base <ref> or whether another projects has been changed the project depends on via a path: ../other_package dependency.
• Both dependencies and dev_dependencies are considered for the dependency.

Additional Context

This is what tests look like currently for us ...

They run for 20 minutes and emit > 6000 log lines, while not showcasing the test failures at the very end:

Start by locating the CLI command registration and the existing workspace or package-discovery tests, then trace how pubspec.yaml dependencies and git diffs are handled. Done means the new workspace matrix command accepts --base, returns each affected project's name and relative path as JSON, follows path dependencies from dependencies and dev_dependencies, and keeps CI checks and coverage passing.