mfogliatto / mfogliatto/ReferenceCop

Add automated end-to-end testing using playground in CI/CD workflows

Open
#43 0 comments 0 reactions 0 assignees View on GitHub
chore help wanted
Dominant language
C#
Stars
1
Forks
2
PR merge metrics
No merged PRs in 30d

Description

## Overview
Extend the playground project to support automated end-to-end testing that validates ReferenceCop's key scenarios in PR and official build workflows.

## Context
Currently, the playground project in `playground/` provides a manual testing environment where developers can:
- Build the ReferenceCop package
- Test it against sample projects with various rule violations
- Validate changes end-to-end before committing

This works well for manual validation, but we should automate this to ensure all key scenarios are tested on every check-in.

## Goals
1. Create automated test scenarios that validate all ReferenceCop rule types:
- `AssemblyName` rules (e.g., blocking Newtonsoft.Json)
- `ProjectTag` rules (e.g., preventing App projects from referencing Tools)
- `ProjectPath` rules (e.g., preventing App from referencing Internal libraries)

2. Integrate these tests into GitHub Actions workflows:
- Run on every PR build
- Run on official builds
- Fail the build if any scenario doesn't produce expected results

3. Test both positive and negative scenarios:
- Verify violations are correctly detected (build fails with expected diagnostics)
- Verify valid references are allowed (build succeeds)

## Implementation Approach

### Option 1: PowerShell Test Runner
Create a test runner script that:
- Builds various test configurations in the playground
- Validates expected build outcomes (success/failure)
- Checks for expected diagnostic messages
- Returns appropriate exit codes for CI/CD

### Option 2: Integration Test Project
Create a proper MSTest/xUnit test project that:
- Uses the playground projects as test fixtures
- Invokes MSBuild programmatically
- Asserts on build results and diagnostics
- Integrates with existing test infrastructure

### Option 3: Combination
- Use PowerShell for simple scenario validation
- Add integration tests for complex validation logic
- Both run in CI/CD pipeline

## Acceptance Criteria
- [ ] Automated tests validate all three rule types (AssemblyName, ProjectTag, ProjectPath)
- [ ] Tests verify both violations (build failure) and valid scenarios (build success)
- [ ] Tests check for expected diagnostic codes (RC0001, RC0002) and messages
- [ ] Tests run in PR build workflow (`.github/workflows/pr-build.yaml` or similar)
- [ ] Tests run in official build workflow (`.github/workflows/official-build.yaml`)
- [ ] Failed tests cause the CI/CD build to fail
- [ ] Test results are visible in GitHub Actions UI
- [ ] Documentation updated with automated testing approach

## Benefits
- Catch regressions early in the development cycle
- Ensure all rule types work correctly across releases
- Provide confidence when accepting community contributions
- Validate package integration scenarios automatically
- Reduce manual testing effort

## Related
- Playground: `playground/`
- Test scenarios reference: `playground/TEST-SCENARIOS.md`
- Build workflows: `.github/workflows/`

Contributor guide

Open the contributing guide

Research direction

Start by reading playground/TEST-SCENARIOS.md and inspecting the existing files under playground/ and .github/workflows/. Compare the PR and official workflow entry points before choosing between a PowerShell runner, an integration test project, or both. Done means all three rule types have positive and negative scenarios, expected diagnostics are checked, workflow failures are visible, and the testing documentation is updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, github-actions
Domain
build-system, ci-cd, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.