enhancement: E2E verify workflow with cross-platform matrix testing
- Dominant language
- F#
- Stars
- 14
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Description
# Enhancement: E2E Verify Workflow with Cross-Platform Matrix Testing
## Summary
Create a comprehensive E2E verification system that can be invoked manually or by the QA Tester skill to run end-to-end tests across multiple platforms before merging PRs.
## Motivation
Currently, E2E tests only run on the local platform during development. This can miss platform-specific issues that only surface on specific OS/architecture combinations. We need:
1. **Manual E2E verification** before merging critical PRs
2. **Cross-platform testing** to catch platform-specific bugs
3. **QA Tester skill integration** for automated pre-merge verification
4. **Platform health monitoring** to track which platforms are fully supported
## Proposed Solution
### 1. GitHub Actions Workflow: `.github/workflows/e2e-verify.yml`
Create a manually-triggered workflow with matrix testing:
\`\`\`yaml
name: E2E Verify
on:
workflow_dispatch:
inputs:
platform_mode:
description: 'Platform testing mode'
required: true
type: choice
options:
- 'supported' # Only test known supported platforms
- 'all' # Test all platform combinations
- 'local' # Placeholder - tests run on triggering platform
executable_types:
description: 'Executable types to test'
required: true
type: choice
default: 'platform-appropriate'
options:
- 'platform-appropriate' # Auto-detect based on platform
- 'framework-dependent'
- 'self-contained'
- 'all'
jobs:
e2e-matrix:
strategy:
matrix:
# Supported platforms (known to work)
os: [ubuntu-latest, windows-latest, macos-latest]
arch: [x64]
include:
- os: macos-latest
arch: arm64
# All platforms mode includes experimental
${{ if inputs.platform_mode == 'all' }}:
include:
- os: ubuntu-latest
arch: arm64
- os: windows-latest
arch: arm64
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- name: Setup .NET
uses: actions/setup-dotnet@v4
with:
dotnet-version: '10.0.x'
- name: Run E2E Tests
run: |
./build.sh TestE2E --executable-type=${{ inputs.executable_types }}
- name: Upload Test Results
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-results-${{ matrix.os }}-${{ matrix.arch }}
path: artifacts/test-results/
\`\`\`
### 2. F# Automation Script: \`e2e-verify.fsx\`
Create \`.claude/skills/qa-tester/scripts/e2e-verify.fsx\`:
\`\`\`fsharp
#!/usr/bin/env dotnet fsi
open System
open System.Diagnostics
type ExecutionMode =
| Local
| RemoteSupported
| RemoteAll
type ExecutableTypes =
| PlatformAppropriate
| FrameworkDependent
| SelfContained
| All
let promptExecutionMode() =
printfn "E2E Verification Mode:"
printfn " 1. Local only (current platform)"
printfn " 2. Remote - supported platforms (ubuntu-x64, windows-x64, macos-x64, macos-arm64)"
printfn " 3. Remote - all platforms (includes experimental ARM64)"
printf "Choose mode [1-3]: "
match Console.ReadLine() with
| "1" -> Local
| "2" -> RemoteSupported
| "3" -> RemoteAll
| _ ->
printfn "Invalid choice. Defaulting to local."
Local
let promptExecutableTypes mode =
match mode with
| Local ->
printfn "\nExecutable types to test:"
printfn " 1. Platform-appropriate (auto-detect)"
printfn " 2. Framework-dependent only"
printfn " 3. Self-contained only"
printfn " 4. All types (may fail on unsupported platforms)"
printf "Choose types [1-4]: "
match Console.ReadLine() with
| "1" -> PlatformAppropriate
| "2" -> FrameworkDependent
| "3" -> SelfContained
| "4" -> All
| _ -> PlatformAppropriate
| _ -> PlatformAppropriate // Always use platform-appropriate for remote
let runLocal executableTypes =
printfn "\nπ Running E2E tests locally..."
let execTypeArg =
match executableTypes with
| PlatformAppropriate -> "platform-appropriate"
| FrameworkDependent -> "framework-dependent"
| SelfContained -> "self-contained"
| All -> "all"
let psi = ProcessStartInfo()
psi.FileName <- "./build.sh"
psi.Arguments <- $"TestE2E --executable-type={execTypeArg}"
psi.UseShellExecute <- false
use proc = Process.Start(psi)
proc.WaitForExit()
if proc.ExitCode = 0 then
printfn "β
Local E2E tests PASSED"
else
printfn "β Local E2E tests FAILED"
exit proc.ExitCode
let triggerRemoteWorkflow platformMode executableTypes =
printfn "\nπ Triggering remote E2E verification workflow..."
let platformArg =
match platformMode with
| RemoteSupported -> "supported"
| RemoteAll -> "all"
| _ -> "supported"
let execTypeArg =
match executableTypes with
| PlatformAppropriate -> "platform-appropriate"
| FrameworkDependent -> "framework-dependent"
| SelfContained -> "self-contained"
| All -> "all"
let psi = ProcessStartInfo()
psi.FileName <- "gh"
psi.Arguments <- $"workflow run e2e-verify.yml -f platform_mode={platformArg} -f executable_types={execTypeArg}"
psi.UseShellExecute <- false
use proc = Process.Start(psi)
proc.WaitForExit()
if proc.ExitCode = 0 then
printfn "β
Workflow triggered successfully"
printfn "π Monitor progress: gh run list --workflow=e2e-verify.yml"
printfn "π Watch run: gh run watch"
else
printfn "β Failed to trigger workflow"
exit proc.ExitCode
// Main execution
printfn "βββββββββββββββββββββββββββββββββββββββ"
printfn " E2E Verification Tool"
printfn "βββββββββββββββββββββββββββββββββββββββ\n"
let mode = promptExecutionMode()
let execTypes = promptExecutableTypes mode
match mode with
| Local -> runLocal execTypes
| RemoteSupported -> triggerRemoteWorkflow RemoteSupported execTypes
| RemoteAll -> triggerRemoteWorkflow RemoteAll execTypes
printfn "\nβ
E2E verification complete"
\`\`\`
### 3. QA Tester Skill Integration
Update \`.claude/skills/qa-tester/SKILL.md\`:
#### New Playbook: E2E Cross-Platform Verification
**When**: Before merging critical PRs or releases
**Steps**:
**Phase 1: Choose Verification Scope**
1. Determine if local-only testing is sufficient
2. For critical changes, opt for cross-platform remote testing
3. Choose platform mode:
- \`supported\`: Test on known working platforms (recommended)
- \`all\`: Include experimental platforms (comprehensive)
**Phase 2: Execute Verification**
1. Run automation script:
\`\`\`bash
dotnet fsi .claude/skills/qa-tester/scripts/e2e-verify.fsx
\`\`\`
2. Follow interactive prompts
3. For remote runs, monitor workflow progress
**Phase 3: Analyze Results**
1. Review test results from all platforms
2. Identify platform-specific failures
3. Verify all critical platforms pass
4. Document any platform limitations
#### New Automation Script
**e2e-verify.fsx**
- Interactive prompt for execution mode (local/remote)
- Platform selection (supported/all)
- Executable type selection
- Triggers GitHub Actions workflow for remote runs
- **Token Savings**: ~800 tokens (vs manual workflow triggering and monitoring)
## Supported Platform Matrix
### Tier 1: Fully Supported (Always Tested)
- Linux x64 (\`ubuntu-latest\`)
- Windows x64 (\`windows-latest\`)
- macOS x64 (\`macos-13\`)
- macOS ARM64 (\`macos-latest\`)
### Tier 2: Experimental (Tested in 'All' Mode)
- Linux ARM64 (self-hosted or future ubuntu-arm64)
- Windows ARM64 (self-hosted or future windows-arm64)
## Benefits
1. **Catch platform-specific bugs** before merging
2. **Manual verification** for critical PRs
3. **QA Tester automation** for consistent testing
4. **Platform health tracking** - know what works where
5. **Flexible testing** - local or remote, supported or all
## Integration with Existing Skills
### QA Tester Skill
- Add E2E cross-platform verification playbook
- Add \`e2e-verify.fsx\` automation script
- Document in SKILL.md and README.md
### Release Manager Skill
- Run E2E verification before release deployment
- Validate all supported platforms pass
### AOT Guru Skill
- Coordinate on platform-specific AOT testing
- Share platform support matrix
## Implementation Phases
### Phase 1: Core Infrastructure
- [ ] Create \`.github/workflows/e2e-verify.yml\`
- [ ] Implement basic matrix testing for supported platforms
- [ ] Test workflow execution manually
### Phase 2: Automation Script
- [ ] Create \`e2e-verify.fsx\`
- [ ] Implement interactive prompts
- [ ] Add GitHub CLI integration
- [ ] Test local and remote modes
### Phase 3: QA Tester Integration
- [ ] Update QA Tester SKILL.md with new playbook
- [ ] Add script to \`.claude/skills/qa-tester/scripts/\`
- [ ] Document usage examples
- [ ] Add to skills-reference.md
### Phase 4: Documentation & Testing
- [ ] Update main README with E2E verification info
- [ ] Document platform support matrix
- [ ] Test complete workflow end-to-end
- [ ] Gather feedback and refine
## Acceptance Criteria
- [ ] GitHub Actions workflow created and tested
- [ ] Workflow supports manual trigger with inputs
- [ ] Matrix testing works for supported platforms
- [ ] \`e2e-verify.fsx\` script created and functional
- [ ] Script provides interactive prompts
- [ ] Script can trigger remote workflow via GitHub CLI
- [ ] QA Tester skill documentation updated
- [ ] Platform support matrix documented
- [ ] End-to-end testing validates all modes work
- [ ] Script integrated into QA Tester skill directory
## Related Issues
- #264 - Platform-aware AOT testing (dependency)
- Related to QA Tester skill enhancements
- Related to release management workflow
## Labels
- enhancement
- qa-tester
- infrastructure
- agent-guidance
Contributor guide
Assessment
This issue has not been assessed yet.