atxtechbro / atxtechbro/dotfiles
feat(ci): add GitHub Actions CI/CD for MCP server protocol testing
- Dominant language
- Shell
- Stars
- 27
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Add automated CI/CD testing for MCP servers using the protocol-level smoke tests we've developed and validated.
## Background
We've developed and battle-tested MCP protocol smoke tests that verify actual JSON-RPC communication vs just CLI wrapper testing. These tests catch real MCP communication failures and should be automated in CI/CD.
## Reference Implementation
Our smoke test procedure is documented in:
- [MCP README - Protocol Testing](mcp/README.md#protocol-testing)
- [Rules procedure](~/.amazonq/rules/procedures/mcp-protocol-smoke-test.md)
## Proposed GitHub Actions Workflow
Create `.github/workflows/mcp-server-tests.yml` that:
### 1. **Test Matrix**
- Test all MCP servers in `mcp/servers/` directory
- Run on multiple Python versions (3.10, 3.11, 3.12)
- Test on ubuntu-latest (primary) and macos-latest
### 2. **MCP Protocol Smoke Tests**
For each MCP server, run the complete handshake:
```bash
(echo '{"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {"protocolVersion": "2024-11-05", "capabilities": {}, "clientInfo": {"name": "ci-smoke-test", "version": "1.0.0"}}}'; echo '{"jsonrpc": "2.0", "method": "notifications/initialized"}'; echo '{"jsonrpc": "2.0", "id": 2, "method": "tools/list"}') | .venv/bin/python -m mcp_server_git -r .
```
### 3. **Test Validation**
- ✅ **Success criteria**:
- Initialize returns protocol version and capabilities
- tools/list returns tool definitions with schemas
- No JSON-RPC error responses
- Exit code 0
- ❌ **Failure detection**:
- Missing initialization errors
- Malformed JSON responses
- Server crashes or silent exits
- Wrong protocol responses
### 4. **Negative Test Cases**
Also test expected failure modes:
- Skip initialization → should fail with clear error
- Wrong method names → should fail with validation errors
- Malformed JSON → should exit cleanly
## Benefits
- **Automated validation** of MCP server functionality after changes
- **Prevents regressions** during subtraction/refactoring work
- **Protocol-level confidence** vs just CLI testing
- **Tracer bullets principle** - immediate feedback on actual functionality
- **Documentation validation** - proves our smoke test guidance works
## Implementation Notes
- Use existing virtual environments in each server directory
- Set up proper git repository context for git-mcp-server
- Handle Docker-based servers appropriately
- Consider secrets/authentication for servers that need it (maybe skip those in CI)
## Acceptance Criteria
- [ ] GitHub Actions workflow runs on PR and push to main
- [ ] Tests all MCP servers with protocol-level smoke tests
- [ ] Validates both positive and negative test cases
- [ ] Provides clear failure messages when tests fail
- [ ] Runs efficiently (parallel where possible)
- [ ] Documents any servers that can't be tested in CI (due to secrets/Docker/etc.)
This embodies the **tracer bullets principle** by providing automated, immediate feedback on MCP server functionality without manual intervention.
Contributor guide
No contributing guide indexed for this repository
Research direction
Read mcp/README.md#protocol-testing and inspect the servers under mcp/ first. Use the proposed .github/workflows/mcp-server-tests.yml and the documented mcp_server_git smoke-test command as entry points. Done means the workflow covers the stated Python versions, platforms, positive and negative protocol cases, and documents servers excluded from CI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ci-cd, devops, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100