finos / finos/morphir-dotnet

enhancement: E2E verify workflow with cross-platform matrix testing

Open
#265 0 comments 0 reactions 0 assignees View on GitHub
agent-guidance enhancement infrastructure
Dominant language
F#
Stars
14
Forks
12
PR merge metrics
No merged PRs in 30d

Description

# Enhancement: E2E Verify Workflow with Cross-Platform Matrix Testing

## Summary

Create a comprehensive E2E verification system that can be invoked manually or by the QA Tester skill to run end-to-end tests across multiple platforms before merging PRs.

## Motivation

Currently, E2E tests only run on the local platform during development. This can miss platform-specific issues that only surface on specific OS/architecture combinations. We need:

1. **Manual E2E verification** before merging critical PRs
2. **Cross-platform testing** to catch platform-specific bugs
3. **QA Tester skill integration** for automated pre-merge verification
4. **Platform health monitoring** to track which platforms are fully supported

## Proposed Solution

### 1. GitHub Actions Workflow: `.github/workflows/e2e-verify.yml`

Create a manually-triggered workflow with matrix testing:

\`\`\`yaml
name: E2E Verify

on:
workflow_dispatch:
inputs:
platform_mode:
description: 'Platform testing mode'
required: true
type: choice
options:
- 'supported' # Only test known supported platforms
- 'all' # Test all platform combinations
- 'local' # Placeholder - tests run on triggering platform
executable_types:
description: 'Executable types to test'
required: true
type: choice
default: 'platform-appropriate'
options:
- 'platform-appropriate' # Auto-detect based on platform
- 'framework-dependent'
- 'self-contained'
- 'all'

jobs:
e2e-matrix:
strategy:
matrix:
# Supported platforms (known to work)
os: [ubuntu-latest, windows-latest, macos-latest]
arch: [x64]
include:
- os: macos-latest
arch: arm64
# All platforms mode includes experimental
${{ if inputs.platform_mode == 'all' }}:
include:
- os: ubuntu-latest
arch: arm64
- os: windows-latest
arch: arm64

runs-on: ${{ matrix.os }}

steps:
- uses: actions/checkout@v4

- name: Setup .NET
uses: actions/setup-dotnet@v4
with:
dotnet-version: '10.0.x'

- name: Run E2E Tests
run: |
./build.sh TestE2E --executable-type=${{ inputs.executable_types }}

- name: Upload Test Results
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-results-${{ matrix.os }}-${{ matrix.arch }}
path: artifacts/test-results/
\`\`\`

### 2. F# Automation Script: \`e2e-verify.fsx\`

Create \`.claude/skills/qa-tester/scripts/e2e-verify.fsx\`:

\`\`\`fsharp
#!/usr/bin/env dotnet fsi

open System
open System.Diagnostics

type ExecutionMode =
| Local
| RemoteSupported
| RemoteAll

type ExecutableTypes =
| PlatformAppropriate
| FrameworkDependent
| SelfContained
| All

let promptExecutionMode() =
printfn "E2E Verification Mode:"
printfn " 1. Local only (current platform)"
printfn " 2. Remote - supported platforms (ubuntu-x64, windows-x64, macos-x64, macos-arm64)"
printfn " 3. Remote - all platforms (includes experimental ARM64)"
printf "Choose mode [1-3]: "

match Console.ReadLine() with
| "1" -> Local
| "2" -> RemoteSupported
| "3" -> RemoteAll
| _ ->
printfn "Invalid choice. Defaulting to local."
Local

let promptExecutableTypes mode =
match mode with
| Local ->
printfn "\nExecutable types to test:"
printfn " 1. Platform-appropriate (auto-detect)"
printfn " 2. Framework-dependent only"
printfn " 3. Self-contained only"
printfn " 4. All types (may fail on unsupported platforms)"
printf "Choose types [1-4]: "

match Console.ReadLine() with
| "1" -> PlatformAppropriate
| "2" -> FrameworkDependent
| "3" -> SelfContained
| "4" -> All
| _ -> PlatformAppropriate
| _ -> PlatformAppropriate // Always use platform-appropriate for remote

let runLocal executableTypes =
printfn "\nπŸ” Running E2E tests locally..."

let execTypeArg =
match executableTypes with
| PlatformAppropriate -> "platform-appropriate"
| FrameworkDependent -> "framework-dependent"
| SelfContained -> "self-contained"
| All -> "all"

let psi = ProcessStartInfo()
psi.FileName <- "./build.sh"
psi.Arguments <- $"TestE2E --executable-type={execTypeArg}"
psi.UseShellExecute <- false

use proc = Process.Start(psi)
proc.WaitForExit()

if proc.ExitCode = 0 then
printfn "βœ… Local E2E tests PASSED"
else
printfn "❌ Local E2E tests FAILED"
exit proc.ExitCode

let triggerRemoteWorkflow platformMode executableTypes =
printfn "\n🌐 Triggering remote E2E verification workflow..."

let platformArg =
match platformMode with
| RemoteSupported -> "supported"
| RemoteAll -> "all"
| _ -> "supported"

let execTypeArg =
match executableTypes with
| PlatformAppropriate -> "platform-appropriate"
| FrameworkDependent -> "framework-dependent"
| SelfContained -> "self-contained"
| All -> "all"

let psi = ProcessStartInfo()
psi.FileName <- "gh"
psi.Arguments <- $"workflow run e2e-verify.yml -f platform_mode={platformArg} -f executable_types={execTypeArg}"
psi.UseShellExecute <- false

use proc = Process.Start(psi)
proc.WaitForExit()

if proc.ExitCode = 0 then
printfn "βœ… Workflow triggered successfully"
printfn "πŸ“Š Monitor progress: gh run list --workflow=e2e-verify.yml"
printfn "πŸ“Š Watch run: gh run watch"
else
printfn "❌ Failed to trigger workflow"
exit proc.ExitCode

// Main execution
printfn "═══════════════════════════════════════"
printfn " E2E Verification Tool"
printfn "═══════════════════════════════════════\n"

let mode = promptExecutionMode()
let execTypes = promptExecutableTypes mode

match mode with
| Local -> runLocal execTypes
| RemoteSupported -> triggerRemoteWorkflow RemoteSupported execTypes
| RemoteAll -> triggerRemoteWorkflow RemoteAll execTypes

printfn "\nβœ… E2E verification complete"
\`\`\`

### 3. QA Tester Skill Integration

Update \`.claude/skills/qa-tester/SKILL.md\`:

#### New Playbook: E2E Cross-Platform Verification

**When**: Before merging critical PRs or releases

**Steps**:

**Phase 1: Choose Verification Scope**
1. Determine if local-only testing is sufficient
2. For critical changes, opt for cross-platform remote testing
3. Choose platform mode:
- \`supported\`: Test on known working platforms (recommended)
- \`all\`: Include experimental platforms (comprehensive)

**Phase 2: Execute Verification**
1. Run automation script:
\`\`\`bash
dotnet fsi .claude/skills/qa-tester/scripts/e2e-verify.fsx
\`\`\`
2. Follow interactive prompts
3. For remote runs, monitor workflow progress

**Phase 3: Analyze Results**
1. Review test results from all platforms
2. Identify platform-specific failures
3. Verify all critical platforms pass
4. Document any platform limitations

#### New Automation Script

**e2e-verify.fsx**
- Interactive prompt for execution mode (local/remote)
- Platform selection (supported/all)
- Executable type selection
- Triggers GitHub Actions workflow for remote runs
- **Token Savings**: ~800 tokens (vs manual workflow triggering and monitoring)

## Supported Platform Matrix

### Tier 1: Fully Supported (Always Tested)
- Linux x64 (\`ubuntu-latest\`)
- Windows x64 (\`windows-latest\`)
- macOS x64 (\`macos-13\`)
- macOS ARM64 (\`macos-latest\`)

### Tier 2: Experimental (Tested in 'All' Mode)
- Linux ARM64 (self-hosted or future ubuntu-arm64)
- Windows ARM64 (self-hosted or future windows-arm64)

## Benefits

1. **Catch platform-specific bugs** before merging
2. **Manual verification** for critical PRs
3. **QA Tester automation** for consistent testing
4. **Platform health tracking** - know what works where
5. **Flexible testing** - local or remote, supported or all

## Integration with Existing Skills

### QA Tester Skill
- Add E2E cross-platform verification playbook
- Add \`e2e-verify.fsx\` automation script
- Document in SKILL.md and README.md

### Release Manager Skill
- Run E2E verification before release deployment
- Validate all supported platforms pass

### AOT Guru Skill
- Coordinate on platform-specific AOT testing
- Share platform support matrix

## Implementation Phases

### Phase 1: Core Infrastructure
- [ ] Create \`.github/workflows/e2e-verify.yml\`
- [ ] Implement basic matrix testing for supported platforms
- [ ] Test workflow execution manually

### Phase 2: Automation Script
- [ ] Create \`e2e-verify.fsx\`
- [ ] Implement interactive prompts
- [ ] Add GitHub CLI integration
- [ ] Test local and remote modes

### Phase 3: QA Tester Integration
- [ ] Update QA Tester SKILL.md with new playbook
- [ ] Add script to \`.claude/skills/qa-tester/scripts/\`
- [ ] Document usage examples
- [ ] Add to skills-reference.md

### Phase 4: Documentation & Testing
- [ ] Update main README with E2E verification info
- [ ] Document platform support matrix
- [ ] Test complete workflow end-to-end
- [ ] Gather feedback and refine

## Acceptance Criteria

- [ ] GitHub Actions workflow created and tested
- [ ] Workflow supports manual trigger with inputs
- [ ] Matrix testing works for supported platforms
- [ ] \`e2e-verify.fsx\` script created and functional
- [ ] Script provides interactive prompts
- [ ] Script can trigger remote workflow via GitHub CLI
- [ ] QA Tester skill documentation updated
- [ ] Platform support matrix documented
- [ ] End-to-end testing validates all modes work
- [ ] Script integrated into QA Tester skill directory

## Related Issues

- #264 - Platform-aware AOT testing (dependency)
- Related to QA Tester skill enhancements
- Related to release management workflow

## Labels

- enhancement
- qa-tester
- infrastructure
- agent-guidance

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.