atxtechbro / atxtechbro/dotfiles

feat(ci): add GitHub Actions CI/CD for MCP server protocol testing

Open
#457 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
27
Forks
2
PR merge metrics
No merged PRs in 30d

Description

## Summary

Add automated CI/CD testing for MCP servers using the protocol-level smoke tests we've developed and validated.

## Background

We've developed and battle-tested MCP protocol smoke tests that verify actual JSON-RPC communication vs just CLI wrapper testing. These tests catch real MCP communication failures and should be automated in CI/CD.

## Reference Implementation

Our smoke test procedure is documented in:
- [MCP README - Protocol Testing](mcp/README.md#protocol-testing)
- [Rules procedure](~/.amazonq/rules/procedures/mcp-protocol-smoke-test.md)

## Proposed GitHub Actions Workflow

Create `.github/workflows/mcp-server-tests.yml` that:

### 1. **Test Matrix**
- Test all MCP servers in `mcp/servers/` directory
- Run on multiple Python versions (3.10, 3.11, 3.12)
- Test on ubuntu-latest (primary) and macos-latest

### 2. **MCP Protocol Smoke Tests**
For each MCP server, run the complete handshake:
```bash
(echo '{"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {"protocolVersion": "2024-11-05", "capabilities": {}, "clientInfo": {"name": "ci-smoke-test", "version": "1.0.0"}}}'; echo '{"jsonrpc": "2.0", "method": "notifications/initialized"}'; echo '{"jsonrpc": "2.0", "id": 2, "method": "tools/list"}') | .venv/bin/python -m mcp_server_git -r .
```

### 3. **Test Validation**
- ✅ **Success criteria**:
- Initialize returns protocol version and capabilities
- tools/list returns tool definitions with schemas
- No JSON-RPC error responses
- Exit code 0
- ❌ **Failure detection**:
- Missing initialization errors
- Malformed JSON responses
- Server crashes or silent exits
- Wrong protocol responses

### 4. **Negative Test Cases**
Also test expected failure modes:
- Skip initialization → should fail with clear error
- Wrong method names → should fail with validation errors
- Malformed JSON → should exit cleanly

## Benefits

- **Automated validation** of MCP server functionality after changes
- **Prevents regressions** during subtraction/refactoring work
- **Protocol-level confidence** vs just CLI testing
- **Tracer bullets principle** - immediate feedback on actual functionality
- **Documentation validation** - proves our smoke test guidance works

## Implementation Notes

- Use existing virtual environments in each server directory
- Set up proper git repository context for git-mcp-server
- Handle Docker-based servers appropriately
- Consider secrets/authentication for servers that need it (maybe skip those in CI)

## Acceptance Criteria

- [ ] GitHub Actions workflow runs on PR and push to main
- [ ] Tests all MCP servers with protocol-level smoke tests
- [ ] Validates both positive and negative test cases
- [ ] Provides clear failure messages when tests fail
- [ ] Runs efficiently (parallel where possible)
- [ ] Documents any servers that can't be tested in CI (due to secrets/Docker/etc.)

This embodies the **tracer bullets principle** by providing automated, immediate feedback on MCP server functionality without manual intervention.

Contributor guide

No contributing guide indexed for this repository

Research direction

Read mcp/README.md#protocol-testing and inspect the servers under mcp/ first. Use the proposed .github/workflows/mcp-server-tests.yml and the documented mcp_server_git smoke-test command as entry points. Done means the workflow covers the stated Python versions, platforms, positive and negative protocol cases, and documents servers excluded from CI.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, devops, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.