lnp2pBot / lnp2pBot/bot

Add mutation testing with StrykerJS to evaluate test suite quality

Open
#760 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
TypeScript
Stars
292
Forks
136
Avg merge
19m
Merged PRs (30d)
1

Description

## What is Mutation Testing?

Mutation testing is a technique that evaluates the **quality of your test suite** by introducing small changes (mutations) to the source code and checking if the tests detect them.

### How it works

1. A mutation tool takes your source code and creates small variations called **mutants** — for example:
- Changing `>` to `>=`
- Replacing `true` with `false`
- Removing a function call
- Changing `+` to `-`
- Replacing a string with an empty string

2. For each mutant, the tool runs your test suite
3. If the tests **fail** → the mutant was **killed** ✅ (tests caught the change)
4. If the tests **pass** → the mutant **survived** ❌ (tests missed the change)

A high **mutation score** (% of killed mutants) means your tests are effective at catching bugs. A low score means there are code paths where bugs could hide undetected.

### Why it matters

Code coverage only tells you which lines were **executed** during tests — not whether the tests actually **verify correct behavior**. A test that calls a function but never asserts its return value gives 100% coverage but catches 0% of bugs.

Mutation testing answers a deeper question: **"If a bug were introduced here, would our tests catch it?"**

### Concrete example from this codebase

Consider `util/index.ts` — functions like `getBtcFiatPrice()`, `numberFormat()`, `getCurrency()`, etc. A mutation tool might:

| Original | Mutant | What it tests |
|----------|--------|---------------|
| `if (amount > 0)` | `if (amount >= 0)` | Does the test check the zero boundary? |
| `return rate * amount` | `return rate / amount` | Does the test verify the calculation result? |
| `user.banned = true` | `user.banned = false` | Does the test assert the ban actually applied? |
| `order.status = 'ACTIVE'` | `order.status = ''` | Does the test check the status value? |

If any of these mutants survive, it means a real bug in that code path would also go undetected.

## Current Test Coverage

The project has **5 test files** covering **70+ source files**:

| Test file | What it covers |
|-----------|---------------|
| `tests/bot/bot.spec.ts` | Bot commands |
| `tests/bot/validation.spec.ts` | Input validation |
| `tests/ln/lightning.spec.ts` | Lightning operations |
| `tests/monitoring.spec.ts` | Monitoring module |
| `tests/util/index.spec.ts` | Utility functions |

This is a good starting point, but mutation testing would reveal which of these tests are truly effective at catching bugs vs. just executing code paths.

## Recommended Tool: StrykerJS

[StrykerJS](https://stryker-mutator.io/) is the most mature mutation testing framework for JavaScript/TypeScript projects. It supports:

- ✅ TypeScript (first-class support)
- ✅ Mocha test runner (our current setup)
- ✅ Incremental mode (only test changed files)
- ✅ HTML reports with detailed per-file mutation scores
- ✅ CI integration with dashboard
- ✅ Configurable mutators (choose which mutations to apply)

### Alternatives Considered

| Tool | Verdict |
|------|---------|
| **StrykerJS** | ✅ Best choice — mature, supports TS + Mocha, active community |
| **mutode** | ❌ Abandoned, last release 2019 |
| **mutation-testing-elements** | ❌ Report viewer only, not a test runner |

## Implementation Plan

### Phase 1: Local Setup

1. Install StrykerJS:
```bash
npm install --save-dev @stryker-mutator/core @stryker-mutator/mocha-runner @stryker-mutator/typescript-checker
```

2. Create `stryker.config.json`:
```json
{
"$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
"packageManager": "npm",
"testRunner": "mocha",
"checkers": ["typescript"],
"tsconfigFile": "tsconfig.test.json",
"coverageAnalysis": "perTest",
"mutate": [
"bot/**/*.ts",
"util/**/*.ts",
"ln/**/*.ts",
"models/**/*.ts",
"!**/*.spec.ts",
"!**/mocks/**"
],
"mochaOptions": {
"spec": ["dist/tests/**/*.spec.js"],
"config": ".mocharc.yml"
},
"thresholds": {
"high": 80,
"low": 60,
"break": null
},
"reporters": ["html", "clear-text", "progress"],
"timeoutMS": 60000,
"concurrency": 2
}
```

3. Add npm script:
```json
{
"scripts": {
"mutation-test": "stryker run",
"mutation-test:util": "stryker run --mutate 'util/**/*.ts'"
}
}
```

### Phase 2: CI Workflow (Non-Blocking)

Create `.github/workflows/mutation-testing.yaml`:

```yaml
name: Mutation Testing

on:
push:
branches: [main]
schedule:
- cron: '0 6 * * 1' # Weekly on Monday 6:00 UTC
workflow_dispatch:

jobs:
mutation-test:
runs-on: ubuntu-latest
container:
image: 'ubuntu:24.04'
steps:
- uses: actions/checkout@v4

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'

- name: Setup MongoDB
run: |
apt update && apt install -y sudo curl gnupg
sudo rm -f /etc/ssl/certs/ca-bundle.crt
sudo apt reinstall -y ca-certificates
sudo update-ca-certificates
curl -fsSL https://pgp.mongodb.com/server-6.0.asc | \
sudo gpg -o /usr/share/keyrings/mongodb-server-6.0.gpg --dearmor
echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-6.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/6.0 multiverse" | \
sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list
sudo apt-get update && sudo apt-get install -y mongodb-org
sudo mongod --quiet --config /etc/mongod.conf &
sleep 10
mongosh --eval 'db.getSiblingDB("lnp2pbot").createCollection("mycollection")'

- run: npm ci
- run: npx tsc

- name: Run mutation tests
continue-on-error: true # Non-blocking initially
env:
DB_USER: ''
DB_PASS: ''
DB_HOST: '127.0.0.1'
DB_PORT: '27017'
DB_NAME: 'lnp2pbot'
run: npx stryker run

- name: Upload mutation report
if: always()
uses: actions/upload-artifact@v4
with:
name: mutation-report
path: reports/mutation/
retention-days: 30
```

### Phase 3: Incremental Improvement

Once the baseline is established:

1. Identify modules with lowest mutation scores
2. Write targeted tests for surviving mutants
3. Gradually increase the `thresholds.break` value to enforce minimum mutation score
4. Add incremental mutation testing to PRs (only mutate changed files)

## Suggested Starting Point

Start with `util/` — it has the most isolated, pure functions that are easiest to test thoroughly:
- `util/index.ts` — price calculations, formatting, validation
- `ln/` — Lightning payment logic (critical path)
- `bot/validation.ts` — input validation (already has tests)

These modules have clear inputs/outputs and are the most impactful to verify.

## Benefits Summary

| Benefit | Impact |
|---------|--------|
| **Find weak tests** | Discover tests that execute code without verifying behavior |
| **Prevent regressions** | Ensure critical logic (payments, orders, disputes) is properly guarded |
| **Guide test writing** | Know exactly which code paths need better tests |
| **Confidence in refactoring** | Mutation score tells you if your safety net has holes |
| **Complement coverage** | Coverage = breadth, mutation score = depth |

## References

- [StrykerJS Documentation](https://stryker-mutator.io/docs/stryker-js/introduction/)
- [Mutation Testing on Wikipedia](https://en.wikipedia.org/wiki/Mutation_testing)
- [MostroP2P/mostro#618](https://github.com/MostroP2P/mostro/issues/618) — Same initiative for the Rust codebase
- [grunch/choke#37](https://github.com/grunch/choke/issues/37) — Same initiative for the Flutter codebase

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing package.json, tsconfig.test.json, the existing tests under tests/, and the proposed stryker.config.json settings. Run the current TypeScript build and test suite before trying mutation testing on util/. Done means local mutation reports are generated and the non-blocking .github/workflows/mutation-testing.yaml workflow runs successfully with the documented commands.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, typescript
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.