awslabs / awslabs/cli-agent-orchestrator
Add E2E tests for main web UI (web/) with API mocking
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 267
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 70
Description
## Problem
The main CAO web UI (React app at web/) currently has no E2E test coverage in CI. The web UI is tested only through unit tests (vitest), while cao_mcp_apps/e2e provides E2E coverage for MCP apps but not the main web interface itself.
## Proposed Solution
Add web/e2e Playwright tests for the main web UI with API mocking to provide E2E test coverage without requiring the full backend infrastructure.
### Implementation Plan
1. **Create web/e2e directory structure**
- Create `web/e2e/` directory
- Add Playwright test files for web UI components
- Update `web/playwright.config.ts` to add webServer configuration
2. **Add API mocking**
- Use Playwright's `page.route()` to intercept backend API calls
- Mock `/agents/providers` endpoint to return provider list
- Mock `/agents/profiles` endpoint to return agent profiles
- Mock `/health` endpoint to return {status: "ok"}
- Mock `/sessions` and `/terminals` endpoints for spawn agent tests
3. **Configure webServer in web/playwright.config.ts**
- Add webServer configuration to start the Vite dev server
- Set baseURL to the Vite dev server port (5173)
- Configure timeout and reuseExistingServer options
4. **Add web-e2e job to CI workflow (.github/workflows/ci.yml)**
- Install dependencies
- Install Playwright browsers
- Run tests (Playwright will start the dev server via webServer config)
- Upload test reports as artifacts
### Example Test Structure
```typescript
// web/e2e/web-ui.spec.ts
import { test, expect } from '@playwright/test';
test.describe('Web UI E2E', () => {
test.beforeEach(async ({ page }) => {
// Mock backend APIs
await page.route('**/agents/providers', async route => {
await route.fulfill({
status: 200,
contentType: 'application/json',
body: JSON.stringify([{ name: 'claude_code', binary: 'claude' }]),
});
});
await page.goto('/');
});
test('should load web interface', async ({ page }) => {
await expect(page).toHaveTitle(/Agent Orchestrator/);
});
});
```
### Benefits
- Main web UI gets E2E test coverage in CI
- Tests run without backend infrastructure (faster, simpler CI)
- Tests become pure UI tests, decoupled from backend implementation
- Catches UI regressions before they reach production
### Trade-offs
- Tests don't verify real backend integration
- Need to keep mock responses in sync with actual API contracts
- UI tests won't catch backend API changes
### Alternatives Considered
1. **Start backend server in CI**: Requires significant infrastructure setup (Python, database, dependencies)
2. **Use cao-mcp-apps-e2e pattern**: cao_mcp_apps/e2e exists in main with webServer pattern (builds bundles + starts test harness server), but this is for MCP apps, not the main web UI
3. **No E2E tests for web UI**: Current state
## Out of Scope
- Setting up the full backend API server in CI
- Mocking complex API interactions (websocket connections, streaming responses)
- Integration tests that verify backend behavior
## Related
- cao_mcp_apps/e2e exists in main with webServer pattern (builds bundles + starts test harness server for MCP apps)
- Playwright documentation: https://playwright.dev/docs/mock
Contributor guide
Assessment
This issue has not been assessed yet.