finos / finos/morphir-dotnet

Test skill emulation in GitHub Copilot

Open
#266 0 comments 0 reactions 0 assignees View on GitHub
documentation enhancement
Dominant language
F#
Stars
14
Forks
12
PR merge metrics
No merged PRs in 30d

Description

# Test: Skill Emulation in GitHub Copilot

## Objective

Verify that morphir-dotnet skills (gurus) are accessible and functional in GitHub Copilot through documentation-based emulation, even though Copilot doesn't natively support Claude Code's \`@skill\` command.

## Background

morphir-dotnet provides specialized skills (QA Tester, AOT Guru, Release Manager) designed for Claude Code. While Copilot doesn't support native skill invocation, users should be able to:
1. Discover available skills through documentation
2. Manually invoke skill guidance by reading skill files
3. Run skill automation scripts directly
4. Follow skill playbooks and decision trees

## Test Scope

**Skills to Test**:
- QA Tester (\`.claude/skills/qa-tester/\`)
- AOT Guru (\`.claude/skills/aot-guru/\`)
- Release Manager (\`.claude/skills/release-manager/\`)

**Test Environment**: GitHub Copilot (VS Code extension or CLI)

## Detailed Test Plan

### Phase 1: Skill Discovery

#### Test 1.1: List Available Skills
**User Action**: Ask Copilot "What skills are available in this project?"

**BDD Scenario**:
\`\`\`gherkin
Feature: Skill Discovery in GitHub Copilot

Scenario: User requests list of available skills
Given I am using GitHub Copilot in the morphir-dotnet repository
When I ask "What skills are available in this project?"
Then Copilot should read .agents/skills-reference.md
And Copilot should list "QA Tester", "AOT Guru", and "Release Manager"
And Copilot should provide a description for each skill
And Copilot should reference the skill file locations
\`\`\`

**Acceptance Criteria**:
- [ ] Copilot identifies all 3 skills
- [ ] Provides accurate descriptions
- [ ] References correct file paths

#### Test 1.2: Understanding Skill Aliases
**User Action**: Ask Copilot "Can I use @skill qa instead of @skill qa-tester?"

**BDD Scenario**:
\`\`\`gherkin
Feature: Skill Alias Understanding

Scenario: User asks about skill aliases
Given I am using GitHub Copilot
When I ask "Can I use @skill qa instead of @skill qa-tester?"
Then Copilot should explain aliases are documentation-only
And Copilot should clarify @skill is Claude Code-specific
And Copilot should suggest reading .claude/skills/qa-tester/SKILL.md directly
\`\`\`

**Acceptance Criteria**:
- [ ] Explains alias limitation
- [ ] Clarifies \`@skill\` is Claude-specific
- [ ] Provides alternative approach for Copilot

### Phase 2: Skill Invocation Emulation

#### Test 2.1: Manual Skill Invocation (QA Tester)
**User Action**: Ask Copilot "I want to use the QA Tester skill to create a test plan for PR #123"

**BDD Scenario**:
\`\`\`gherkin
Feature: QA Tester Skill Emulation

Scenario: Create test plan using QA Tester skill
Given I am using GitHub Copilot
And PR #123 exists with acceptance criteria
When I ask "Use the QA Tester skill to create a test plan for PR #123"
Then Copilot should read .claude/skills/qa-tester/SKILL.md
And Copilot should follow the "Test Plan Development" guidance
And Copilot should produce a test plan with:
| Element |
| Happy path scenarios |
| Edge cases |
| Error conditions |
| Priority labels |
| Execution scripts |
\`\`\`

**Acceptance Criteria**:
- [ ] Copilot accesses SKILL.md content
- [ ] Follows skill-specific guidance
- [ ] Produces test plan matching skill template
- [ ] Includes acceptance criteria, test cases, priorities

#### Test 2.2: Running Skill Scripts
**User Action**: Ask Copilot "How do I run the smoke test script from QA Tester?"

**BDD Scenario**:
\`\`\`gherkin
Feature: Skill Script Execution

Scenario: User asks how to run a skill script
Given I am using GitHub Copilot
When I ask "How do I run the smoke test script from QA Tester?"
Then Copilot should reference .claude/skills/qa-tester/scripts/
And Copilot should provide command "dotnet fsi .claude/skills/qa-tester/scripts/smoke-test.fsx"
And Copilot should explain the script validates core functionality
\`\`\`

**Acceptance Criteria**:
- [ ] Identifies correct script location
- [ ] Provides accurate execution command
- [ ] Explains expected behavior

### Phase 3: Skill Guidance Navigation

#### Test 3.1: Following Playbooks
**User Action**: Ask Copilot "Walk me through the regression testing playbook"

**BDD Scenario**:
\`\`\`gherkin
Feature: Playbook Navigation

Scenario: User requests playbook guidance
Given I am using GitHub Copilot
When I ask "Walk me through the regression testing playbook"
Then Copilot should read the QA Tester SKILL.md
And Copilot should identify the "Regression Testing Playbook"
And Copilot should provide steps in order:
| Step | Description |
| 1 | Identify changed areas |
| 2 | Run full test suite |
| 3 | Run E2E tests |
| 4 | Test core user workflows |
| 5 | Check backwards compatibility |
\`\`\`

**Acceptance Criteria**:
- [ ] Locates regression testing playbook
- [ ] Provides sequential steps
- [ ] Includes commands and validation criteria

### Phase 4: Documentation Requirements

To support Copilot skill emulation, AGENTS.md and skills-reference.md must be updated with Copilot-specific guidance.

**Required AGENTS.md Section**:
- How to discover skills in Copilot
- How to invoke skills (alternative to \`@skill\`)
- How to run automation scripts
- Skill documentation locations

**Required skills-reference.md Section**:
- Copilot-specific usage patterns
- Example prompts for skill invocation
- Script execution examples

## Acceptance Criteria

**Functional Requirements**:
- [ ] Copilot can discover all skills via skills-reference.md
- [ ] Copilot can access skill guidance from SKILL.md files
- [ ] Copilot can provide script execution commands
- [ ] Copilot can follow playbooks step-by-step
- [ ] Copilot can apply decision tree logic

**Documentation Requirements**:
- [ ] AGENTS.md includes Copilot-specific section
- [ ] skills-reference.md includes Copilot usage guide
- [ ] Each SKILL.md is Copilot-compatible
- [ ] README references cross-platform skill access

**Quality Requirements**:
- [ ] All BDD scenarios documented
- [ ] Test results captured
- [ ] Issues/limitations identified
- [ ] Workarounds documented

## Deliverables

- [ ] Test execution report (pass/fail for each scenario)
- [ ] Copilot conversation transcripts
- [ ] List of identified issues
- [ ] Documentation improvement suggestions
- [ ] Updated AGENTS.md
- [ ] Updated skills-reference.md

## Success Metrics

- **Discovery**: 100% of skills discoverable
- **Invocation**: 90%+ skill guidance followed correctly
- **Scripts**: 100% of script locations identified
- **Playbooks**: 80%+ playbook steps followed accurately

## Related

- Skills: qa-tester, aot-guru, release-manager
- Documentation: AGENTS.md, skills-reference.md

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.