Add mutation testing with StrykerJS to evaluate test suite quality
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 292
- Forks
- 136
- Avg merge
- 19m
- Merged PRs (30d)
- 1
Description
## What is Mutation Testing?
Mutation testing is a technique that evaluates the **quality of your test suite** by introducing small changes (mutations) to the source code and checking if the tests detect them.
### How it works
1. A mutation tool takes your source code and creates small variations called **mutants** — for example:
- Changing `>` to `>=`
- Replacing `true` with `false`
- Removing a function call
- Changing `+` to `-`
- Replacing a string with an empty string
2. For each mutant, the tool runs your test suite
3. If the tests **fail** → the mutant was **killed** ✅ (tests caught the change)
4. If the tests **pass** → the mutant **survived** ❌ (tests missed the change)
A high **mutation score** (% of killed mutants) means your tests are effective at catching bugs. A low score means there are code paths where bugs could hide undetected.
### Why it matters
Code coverage only tells you which lines were **executed** during tests — not whether the tests actually **verify correct behavior**. A test that calls a function but never asserts its return value gives 100% coverage but catches 0% of bugs.
Mutation testing answers a deeper question: **"If a bug were introduced here, would our tests catch it?"**
### Concrete example from this codebase
Consider `util/index.ts` — functions like `getBtcFiatPrice()`, `numberFormat()`, `getCurrency()`, etc. A mutation tool might:
| Original | Mutant | What it tests |
|----------|--------|---------------|
| `if (amount > 0)` | `if (amount >= 0)` | Does the test check the zero boundary? |
| `return rate * amount` | `return rate / amount` | Does the test verify the calculation result? |
| `user.banned = true` | `user.banned = false` | Does the test assert the ban actually applied? |
| `order.status = 'ACTIVE'` | `order.status = ''` | Does the test check the status value? |
If any of these mutants survive, it means a real bug in that code path would also go undetected.
## Current Test Coverage
The project has **5 test files** covering **70+ source files**:
| Test file | What it covers |
|-----------|---------------|
| `tests/bot/bot.spec.ts` | Bot commands |
| `tests/bot/validation.spec.ts` | Input validation |
| `tests/ln/lightning.spec.ts` | Lightning operations |
| `tests/monitoring.spec.ts` | Monitoring module |
| `tests/util/index.spec.ts` | Utility functions |
This is a good starting point, but mutation testing would reveal which of these tests are truly effective at catching bugs vs. just executing code paths.
## Recommended Tool: StrykerJS
[StrykerJS](https://stryker-mutator.io/) is the most mature mutation testing framework for JavaScript/TypeScript projects. It supports:
- ✅ TypeScript (first-class support)
- ✅ Mocha test runner (our current setup)
- ✅ Incremental mode (only test changed files)
- ✅ HTML reports with detailed per-file mutation scores
- ✅ CI integration with dashboard
- ✅ Configurable mutators (choose which mutations to apply)
### Alternatives Considered
| Tool | Verdict |
|------|---------|
| **StrykerJS** | ✅ Best choice — mature, supports TS + Mocha, active community |
| **mutode** | ❌ Abandoned, last release 2019 |
| **mutation-testing-elements** | ❌ Report viewer only, not a test runner |
## Implementation Plan
### Phase 1: Local Setup
1. Install StrykerJS:
```bash
npm install --save-dev @stryker-mutator/core @stryker-mutator/mocha-runner @stryker-mutator/typescript-checker
```
2. Create `stryker.config.json`:
```json
{
"$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
"packageManager": "npm",
"testRunner": "mocha",
"checkers": ["typescript"],
"tsconfigFile": "tsconfig.test.json",
"coverageAnalysis": "perTest",
"mutate": [
"bot/**/*.ts",
"util/**/*.ts",
"ln/**/*.ts",
"models/**/*.ts",
"!**/*.spec.ts",
"!**/mocks/**"
],
"mochaOptions": {
"spec": ["dist/tests/**/*.spec.js"],
"config": ".mocharc.yml"
},
"thresholds": {
"high": 80,
"low": 60,
"break": null
},
"reporters": ["html", "clear-text", "progress"],
"timeoutMS": 60000,
"concurrency": 2
}
```
3. Add npm script:
```json
{
"scripts": {
"mutation-test": "stryker run",
"mutation-test:util": "stryker run --mutate 'util/**/*.ts'"
}
}
```
### Phase 2: CI Workflow (Non-Blocking)
Create `.github/workflows/mutation-testing.yaml`:
```yaml
name: Mutation Testing
on:
push:
branches: [main]
schedule:
- cron: '0 6 * * 1' # Weekly on Monday 6:00 UTC
workflow_dispatch:
jobs:
mutation-test:
runs-on: ubuntu-latest
container:
image: 'ubuntu:24.04'
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Setup MongoDB
run: |
apt update && apt install -y sudo curl gnupg
sudo rm -f /etc/ssl/certs/ca-bundle.crt
sudo apt reinstall -y ca-certificates
sudo update-ca-certificates
curl -fsSL https://pgp.mongodb.com/server-6.0.asc | \
sudo gpg -o /usr/share/keyrings/mongodb-server-6.0.gpg --dearmor
echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-6.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/6.0 multiverse" | \
sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list
sudo apt-get update && sudo apt-get install -y mongodb-org
sudo mongod --quiet --config /etc/mongod.conf &
sleep 10
mongosh --eval 'db.getSiblingDB("lnp2pbot").createCollection("mycollection")'
- run: npm ci
- run: npx tsc
- name: Run mutation tests
continue-on-error: true # Non-blocking initially
env:
DB_USER: ''
DB_PASS: ''
DB_HOST: '127.0.0.1'
DB_PORT: '27017'
DB_NAME: 'lnp2pbot'
run: npx stryker run
- name: Upload mutation report
if: always()
uses: actions/upload-artifact@v4
with:
name: mutation-report
path: reports/mutation/
retention-days: 30
```
### Phase 3: Incremental Improvement
Once the baseline is established:
1. Identify modules with lowest mutation scores
2. Write targeted tests for surviving mutants
3. Gradually increase the `thresholds.break` value to enforce minimum mutation score
4. Add incremental mutation testing to PRs (only mutate changed files)
## Suggested Starting Point
Start with `util/` — it has the most isolated, pure functions that are easiest to test thoroughly:
- `util/index.ts` — price calculations, formatting, validation
- `ln/` — Lightning payment logic (critical path)
- `bot/validation.ts` — input validation (already has tests)
These modules have clear inputs/outputs and are the most impactful to verify.
## Benefits Summary
| Benefit | Impact |
|---------|--------|
| **Find weak tests** | Discover tests that execute code without verifying behavior |
| **Prevent regressions** | Ensure critical logic (payments, orders, disputes) is properly guarded |
| **Guide test writing** | Know exactly which code paths need better tests |
| **Confidence in refactoring** | Mutation score tells you if your safety net has holes |
| **Complement coverage** | Coverage = breadth, mutation score = depth |
## References
- [StrykerJS Documentation](https://stryker-mutator.io/docs/stryker-js/introduction/)
- [Mutation Testing on Wikipedia](https://en.wikipedia.org/wiki/Mutation_testing)
- [MostroP2P/mostro#618](https://github.com/MostroP2P/mostro/issues/618) — Same initiative for the Rust codebase
- [grunch/choke#37](https://github.com/grunch/choke/issues/37) — Same initiative for the Flutter codebase
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing package.json, tsconfig.test.json, the existing tests under tests/, and the proposed stryker.config.json settings. Run the current TypeScript build and test suite before trying mutation testing on util/. Done means local mutation reports are generated and the non-blocking .github/workflows/mutation-testing.yaml workflow runs successfully with the documented commands.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, typescript
- Domain
- ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100