github / github/spec-kit

Add Automated Testing for Bash and PowerShell Scripts

未关闭
#1,049 3 条评论 2 个 reaction 已指派 0 人 在 GitHub 查看
stale
主要语言
Python
星标
137k
派生
12.3k
平均合并
2 天 7 小时
30 天内合并 PR
155

描述

## Summary

This proposal adds comprehensive automated testing for the shell scripts in `scripts/bash/` and `scripts/powershell/` using Bats and Pester testing frameworks, integrated with GitHub Actions CI.

## Motivation

Currently, the Spec Kit project has no automated tests for its critical shell scripts (common.sh, setup-plan.sh, check-prerequisites.sh, create-new-feature.sh, update-agent-context.sh). This creates risks:

- No regression detection when scripts are modified
- Manual testing burden for contributors
- No verification that Bash and PowerShell scripts produce identical outputs
- Difficult to validate cross-platform compatibility

The CONTRIBUTING.md states "Write tests for new functionality" [1], but the scripts predating this guideline remain untested.

### Related Issues

Several existing issues demonstrate the need for automated testing:

**Script Bugs and Edge Cases**:
- [#1038](https://github.com/github/spec-kit/issues/1038) - Unable to source common.sh when CDPATH is set
- [#1029](https://github.com/github/spec-kit/issues/1029) - Dates generated by scripts are incorrect
- [#975](https://github.com/github/spec-kit/issues/975) - create-new-feature.ps1 ignores existing feature branches when determining next feature number
- [#965](https://github.com/github/spec-kit/issues/965) - Scripts REPO_ROOT variable issue in nested folders
- [#957](https://github.com/github/spec-kit/issues/957) - PowerShell script issue with apostrophes in arguments

**Script Usability Problems**:
- [#842](https://github.com/github/spec-kit/issues/842) - Copilot/Spec-kit seem unable to write PowerShell scripts that work

These issues would have been caught earlier with automated tests, preventing user friction and reducing maintenance burden.

## Proposed Solution

Implement automated testing using industry-standard frameworks:

**For Bash scripts**: Bats (Bash Automated Testing System)
- TAP-compliant testing framework [2]
- Used by major projects (Docker, Kubernetes helpers)
- CI integration with GitHub Actions [3]

**For PowerShell scripts**: Pester
- Official Microsoft-endorsed testing framework [4]
- Pre-installed on GitHub-hosted runners [5]
- Native GitHub Actions support

## Implementation Plan

### Directory Structure
```
tests/
├── bash/
│ ├── test_helper/ # bats-support, bats-assert, bats-file
│ ├── unit/ # Function-level tests
│ └── integration/ # Workflow tests
├── powershell/
│ ├── unit/ # Function-level tests
│ └── integration/ # Workflow tests
└── fixtures/ # Shared test data
```

### Test Coverage Goals

**Unit Tests** (70%+ coverage target):
- `common.sh/ps1`: Core utility functions (get_repo_root, get_current_branch, find_feature_dir_by_prefix)
- `setup-plan.sh/ps1`: Plan template setup, JSON/text output validation
- `check-prerequisites.sh/ps1`: Prerequisite validation logic
- `create-new-feature.sh/ps1`: Branch numbering, naming conventions, Git/non-Git modes

**Integration Tests**:
- `update-agent-context.sh/ps1`: Full workflow including plan parsing, template generation
- Git vs non-Git repository scenarios
- Cross-platform compatibility (Ubuntu, Windows, macOS)
- Parity tests (Bash and PowerShell produce identical outputs)

### CI Integration

GitHub Actions workflow (`.github/workflows/test-scripts.yml`):
- Parallel execution: Bash tests (ubuntu-latest), PowerShell tests (windows-latest)
- Cross-platform smoke tests (all three major OSes)
- Test result artifacts and reports
- Runs on every push/PR affecting scripts or tests

## Benefits

1. **Regression Prevention**: Automated detection of breaking changes
2. **Cross-Platform Validation**: Ensures Windows and Unix scripts stay synchronized
3. **Contributor Confidence**: Contributors can verify their changes don't break existing functionality
4. **Documentation**: Tests serve as executable examples of script usage
5. **Quality Gates**: Aligns with project's "Write tests for new functionality" guideline

## Phased Rollout

To facilitate review, this will be submitted as multiple PRs:

1. PR 1: Test infrastructure and fixtures
2. PR 2: Bash common.sh tests + CI integration
3. PR 3: PowerShell common.ps1 tests + CI integration
4. PR 4: Remaining unit tests (setup-plan, check-prerequisites, create-new-feature)
5. PR 5: Integration tests (update-agent-context, workflows)
6. PR 6: Documentation and CONTRIBUTING.md updates

## Example Test Cases

**Bash (Bats)**:
```bash
@test "get_repo_root returns git root when available" {
cd "$TEST_TEMP_DIR"
git init
source scripts/bash/common.sh

result="$(get_repo_root)"
assert_equal "$result" "$TEST_TEMP_DIR"
}

@test "get_current_branch respects SPECIFY_FEATURE env var" {
export SPECIFY_FEATURE="002-my-feature"
source scripts/bash/common.sh

result="$(get_current_branch)"
assert_equal "$result" "002-my-feature"
}
```

**PowerShell (Pester)**:
```powershell
Describe 'common.ps1' {
It 'Get-RepoRoot returns git root when available' {
Push-Location TestDrive:\
git init
. "$PSScriptRoot/../../../scripts/powershell/common.ps1"

$result = Get-RepoRoot
$result | Should -Be (Get-Location).Path

Pop-Location
}
}
```

## Risks and Mitigations

**Risk**: CI execution time increase
**Mitigation**: Parallel job execution, selective test running on relevant path changes

**Risk**: Flaky tests
**Mitigation**: Proper temporary directory cleanup, timeout configurations, documented retry policies

**Risk**: Maintenance burden
**Mitigation**: Clear test structure, comprehensive documentation, DRY principles with shared fixtures

## Alternative Approaches Considered

1. **Shell script linters only (shellcheck, PSScriptAnalyzer)**
- Rejected: Static analysis doesn't verify runtime behavior

2. **Manual testing checklist**
- Rejected: Error-prone, time-consuming, not scalable

3. **Single test framework for both**
- Rejected: No cross-platform framework provides native feel for both ecosystems

## References

### Documentation
[1] CONTRIBUTING.md - https://github.com/spec-kit/spec-kit/blob/main/CONTRIBUTING.md#submitting-a-pull-request

[2] Bats-core GitHub Repository - https://github.com/bats-core/bats-core

[3] Bats-core Documentation - https://bats-core.readthedocs.io/

[4] Pester Official Website - https://pester.dev/

[5] GitHub Actions: Building and testing PowerShell - https://docs.github.com/en/actions/use-cases-and-examples/building-and-testing/building-and-testing-powershell

[6] Testing Bash with BATS (Opensource.com) - https://opensource.com/article/19/2/testing-bash-bats

[7] PullRequest.com: Testing Bash Scripts with BATS - https://www.pullrequest.com/blog/testing-bash-scripts-with-bats-a-practical-guide/

### Related Issues
[8] Issue #1038 - Unable to source common.sh when CDPATH is set - https://github.com/github/spec-kit/issues/1038

[9] Issue #1029 - Dates generated by scripts are incorrect - https://github.com/github/spec-kit/issues/1029

[10] Issue #987 - create-new-feature shell script is often called incorrectly - https://github.com/github/spec-kit/issues/987

[11] Issue #975 - create-new-feature.ps1 ignores existing feature branches - https://github.com/github/spec-kit/issues/975

[12] Issue #965 - Scripts REPO_ROOT variable issue - https://github.com/github/spec-kit/issues/965

[13] Issue #957 - PowerShell script issue with apostrophes - https://github.com/github/spec-kit/issues/957

[14] Issue #842 - Copilot/Spec-kit seem unable to write PowerShell scripts that work - https://github.com/github/spec-kit/issues/842

[15] Issue #338 - Problems detected by shellcheck on the provided bash scripts - https://github.com/github/spec-kit/issues/338

## AI Assistance Disclosure

This proposal was developed with assistance from [PleaseAI](https://github.com/pleaseai) + Claude Agent SDK (Anthropic). The analysis, test strategy, and implementation plan were collaboratively created. All technical claims have been verified against official documentation, and the proposed approach follows industry best practices for shell script testing.

---

@seon-yunjae

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。