aRustyDev / aRustyDev/mdbook-htmx

docs(adr): ADR-0019: Testing Strategy

Open
#27 0 comments 0 reactions 1 assignee Claimed by @aRustyDev View on GitHub
documentation
Dominant language
Rust
Stars
0
Forks
1
PR merge metrics
No merged PRs in 30d

Description

# ADR-0019: Testing Strategy

## Status

Accepted

## Context

mdbook-htmx is a complex plugin that transforms MDBook output into HTMX-powered documentation. A comprehensive testing strategy is required to ensure correctness, prevent regressions, and maintain quality as the codebase evolves.

Testing challenges include:
- Rust-based renderer with multiple modules
- Tera template rendering
- HTMX-specific behavior
- CSS theming and styling
- Integration with MDBook's build system
- Various deployment targets (static, Cloudflare Workers, Kubernetes)

## Decision Drivers

1. **Confidence** - High test coverage for critical paths
2. **Speed** - Fast feedback during development
3. **Maintenance** - Tests should be easy to update
4. **Realism** - Tests should reflect actual usage
5. **Documentation** - Tests serve as living documentation

## Decision

**Implement a multi-layer testing strategy with unit, integration, snapshot, and end-to-end tests.**

### Test Pyramid

```
╱╲
╱ ╲
╱ E2E╲ Few, slow, high-confidence
╱──────╲
╱Snapshot╲ Template output verification
╱──────────╲
╱ Integration╲ Module interactions
╱──────────────╲
╱ Unit Tests ╲ Fast, isolated, many
╱──────────────────╲
```

### Layer 1: Unit Tests

Test individual functions and modules in isolation.

```rust
// tests/unit/frontmatter_test.rs
use mdbook_htmx::frontmatter::{Frontmatter, parse_frontmatter};

#[test]
fn parse_valid_frontmatter() {
let content = r#"---
title: Test Page
scope: internal
---
# Content"#;

let (frontmatter, body) = parse_frontmatter(content).unwrap();

assert_eq!(frontmatter.title, Some("Test Page".to_string()));
assert_eq!(frontmatter.scope, Some("internal".to_string()));
assert_eq!(body.trim(), "# Content");
}

#[test]
fn parse_empty_frontmatter() {
let content = "# No Frontmatter";
let (frontmatter, body) = parse_frontmatter(content).unwrap();

assert!(frontmatter.is_empty());
assert_eq!(body, content);
}

#[test]
fn parse_invalid_yaml_returns_error() {
let content = r#"---
title: [unclosed
---"#;

let result = parse_frontmatter(content);
assert!(result.is_err());
}
```

```rust
// tests/unit/manifest_test.rs
use mdbook_htmx::manifest::{Manifest, ManifestEntry};

#[test]
fn manifest_version_validation() {
let manifest = Manifest::new("1.0.0");
assert!(manifest.is_compatible_with("1.0.0"));
assert!(manifest.is_compatible_with("1.0.1"));
assert!(!manifest.is_compatible_with("2.0.0"));
}

#[test]
fn manifest_entry_ordering() {
let mut manifest = Manifest::new("1.0.0");
manifest.add_entry(ManifestEntry::new("/guide/intro.html", 2));
manifest.add_entry(ManifestEntry::new("/guide/getting-started.html", 1));

let entries: Vec<_> = manifest.entries().collect();
assert_eq!(entries[0].path, "/guide/getting-started.html");
assert_eq!(entries[1].path, "/guide/intro.html");
}
```

### Layer 2: Integration Tests

Test module interactions and the rendering pipeline.

```rust
// tests/integration/renderer_test.rs
use mdbook_htmx::HtmxRenderer;
use mdbook::MDBook;
use tempfile::tempdir;

#[test]
fn render_simple_book() {
let book_dir = tempdir().unwrap();
setup_test_book(book_dir.path(), TestBookConfig::simple());

let mdbook = MDBook::load(book_dir.path()).unwrap();
let renderer = HtmxRenderer::default();

let result = renderer.render(&mdbook.root, &mdbook.book, &mdbook.config);

assert!(result.is_ok());
assert!(book_dir.path().join("book/index.html").exists());
}

#[test]
fn render_with_frontmatter_scope() {
let book_dir = tempdir().unwrap();
setup_test_book(book_dir.path(), TestBookConfig::with_scopes());

let mdbook = MDBook::load(book_dir.path()).unwrap();
let renderer = HtmxRenderer::default();

renderer.render(&mdbook.root, &mdbook.book, &mdbook.config).unwrap();

// Verify scope is preserved in manifest
let manifest: Manifest = read_manifest(book_dir.path().join("book/manifest.json"));
let internal_page = manifest.get("/internal/secrets.html").unwrap();
assert_eq!(internal_page.scope, Some("internal".to_string()));
}

#[test]
fn render_generates_search_index() {
let book_dir = tempdir().unwrap();
setup_test_book(book_dir.path(), TestBookConfig::with_search());

let mdbook = MDBook::load(book_dir.path()).unwrap();
let renderer = HtmxRenderer::default();

renderer.render(&mdbook.root, &mdbook.book, &mdbook.config).unwrap();

let search_index = book_dir.path().join("book/search-index.json");
assert!(search_index.exists());

let index: SearchIndex = serde_json::from_reader(
std::fs::File::open(&search_index).unwrap()
).unwrap();

assert!(index.pages.len() > 0);
}
```

### Layer 3: Snapshot Tests

Verify template output matches expected results using insta.

```rust
// tests/snapshot/template_test.rs
use insta::assert_snapshot;
use mdbook_htmx::templates::render_page;

#[test]
fn snapshot_page_layout() {
let context = PageContext {
title: "Getting Started".to_string(),
content: "

Welcome to the guide.

".to_string(),
breadcrumbs: vec![
Breadcrumb::new("/", "Home"),
Breadcrumb::new("/guide/", "Guide"),
],
..Default::default()
};

let html = render_page(&context).unwrap();
assert_snapshot!("page_layout", html);
}

#[test]
fn snapshot_navigation() {
let nav = NavigationContext {
chapters: vec![
Chapter::new("Introduction", "/intro.html", false),
Chapter::new("Getting Started", "/getting-started.html", true),
Chapter::new("Advanced", "/advanced.html", false),
],
};

let html = render_navigation(&nav).unwrap();
assert_snapshot!("navigation", html);
}

#[test]
fn snapshot_search_results() {
let results = SearchResults {
query: "authentication".to_string(),
hits: vec![
SearchHit::new("/auth/login.html", "Login", "How to implement authentication..."),
SearchHit::new("/auth/oauth.html", "OAuth", "OAuth authentication flow..."),
],
total: 2,
};

let html = render_search_results(&results).unwrap();
assert_snapshot!("search_results", html);
}

#[test]
fn snapshot_error_pages() {
let contexts = vec![
("401", ErrorContext::unauthorized()),
("403", ErrorContext::forbidden()),
("404", ErrorContext::not_found("/missing")),
("500", ErrorContext::internal_error("DB connection failed")),
];

for (name, ctx) in contexts {
let html = render_error(&ctx).unwrap();
assert_snapshot!(format!("error_{}", name), html);
}
}
```

Snapshot directory structure:
```
tests/snapshots/
├── template_test__page_layout.snap
├── template_test__navigation.snap
├── template_test__search_results.snap
├── template_test__error_401.snap
├── template_test__error_403.snap
├── template_test__error_404.snap
└── template_test__error_500.snap
```

### Layer 4: End-to-End Tests

Test complete workflows including build and serve.

```rust
// tests/e2e/build_test.rs
use assert_cmd::Command;
use predicates::prelude::*;

#[test]
fn build_example_book() {
let mut cmd = Command::cargo_bin("mdbook").unwrap();

cmd.arg("build")
.arg("tests/fixtures/example-book")
.assert()
.success()
.stdout(predicate::str::contains("Rendering with htmx backend"));
}

#[test]
fn build_fails_with_invalid_frontmatter() {
let mut cmd = Command::cargo_bin("mdbook").unwrap();

cmd.arg("build")
.arg("tests/fixtures/invalid-frontmatter")
.assert()
.failure()
.stderr(predicate::str::contains("Invalid frontmatter"));
}

#[test]
fn watch_mode_rebuilds_on_change() {
let book_dir = setup_temp_book();

// Start watch in background
let mut child = Command::cargo_bin("mdbook")
.unwrap()
.arg("watch")
.arg(&book_dir)
.spawn()
.unwrap();

// Wait for initial build
wait_for_file(book_dir.join("book/index.html"), Duration::from_secs(10));

// Modify a file
std::fs::write(book_dir.join("src/chapter.md"), "# Updated").unwrap();

// Wait for rebuild
sleep(Duration::from_millis(500));

let content = std::fs::read_to_string(book_dir.join("book/chapter.html")).unwrap();
assert!(content.contains("Updated"));

child.kill().unwrap();
}
```

### Layer 5: HTMX Behavior Tests

Test HTMX interactions using headless browser (playwright-rust or similar).

```rust
// tests/e2e/htmx_test.rs
use playwright::Playwright;

#[tokio::test]
async fn navigation_uses_htmx_swap() {
let playwright = Playwright::initialize().await.unwrap();
let browser = playwright.chromium().launch(Default::default()).await.unwrap();
let page = browser.new_page(Default::default()).await.unwrap();

// Navigate to docs
page.goto("http://localhost:8080/").await.unwrap();

// Click a navigation link
page.click("a[href='/guide/getting-started.html']").await.unwrap();

// Verify HTMX swapped content (not full page load)
let requests = page.evaluate("window.htmxRequestCount").await.unwrap();
assert!(requests.as_u64().unwrap() > 0);

// Verify URL updated
assert_eq!(page.url().await.unwrap(), "http://localhost:8080/guide/getting-started.html");

// Verify content updated
let heading = page.inner_text("h1").await.unwrap();
assert_eq!(heading, "Getting Started");
}

#[tokio::test]
async fn search_works_with_debounce() {
let page = setup_page().await;
page.goto("http://localhost:8080/").await.unwrap();

// Type in search
page.fill("#search-input", "auth").await.unwrap();

// Wait for debounce
page.wait_for_selector(".search-results", Default::default()).await.unwrap();

// Verify results appear
let results = page.query_selector_all(".search-result").await.unwrap();
assert!(results.len() > 0);
}

#[tokio::test]
async fn theme_toggle_persists() {
let page = setup_page().await;
page.goto("http://localhost:8080/").await.unwrap();

// Toggle to dark theme
page.click("#theme-toggle").await.unwrap();

// Verify localStorage
let theme = page.evaluate("localStorage.getItem('theme')").await.unwrap();
assert_eq!(theme.as_str().unwrap(), "dark");

// Reload and verify persistence
page.reload(Default::default()).await.unwrap();

let body_class = page.get_attribute("body", "class").await.unwrap();
assert!(body_class.unwrap().contains("dark"));
}
```

## Test Fixtures

### Book Fixtures

Create reusable test book configurations:

```rust
// tests/fixtures/mod.rs
pub enum TestBookConfig {
Simple,
WithScopes,
WithSearch,
WithAuth,
Large,
Multilingual,
}

impl TestBookConfig {
pub fn setup(&self, dir: &Path) -> Result<()> {
match self {
TestBookConfig::Simple => setup_simple_book(dir),
TestBookConfig::WithScopes => setup_scoped_book(dir),
// ...
}
}
}

fn setup_simple_book(dir: &Path) -> Result<()> {
fs::write(dir.join("book.toml"), r#"
[book]
title = "Test Book"
authors = ["Test Author"]

[output.htmx]
"#)?;

fs::create_dir_all(dir.join("src"))?;
fs::write(dir.join("src/SUMMARY.md"), r#"
# Summary

- [Introduction](./intro.md)
- [Chapter 1](./chapter1.md)
"#)?;

fs::write(dir.join("src/intro.md"), "# Introduction\n\nWelcome!")?;
fs::write(dir.join("src/chapter1.md"), "# Chapter 1\n\nContent here.")?;

Ok(())
}
```

### Snapshot Fixtures

Store expected outputs:

```
tests/fixtures/
├── books/
│ ├── simple/
│ ├── with-scopes/
│ └── with-search/
└── expected/
├── simple/
│ ├── index.html
│ ├── manifest.json
│ └── search-index.json
└── with-scopes/
└── manifest.json
```

## Configuration

### Cargo.toml

```toml
[dev-dependencies]
# Unit testing
proptest = "1.4"
rstest = "0.18"

# Snapshot testing
insta = { version = "1.34", features = ["yaml", "redactions"] }

# Integration testing
tempfile = "3.9"
assert_cmd = "2.0"
predicates = "3.0"

# E2E testing
playwright = { git = "https://github.com/nicktrav/playwright-rust" }
tokio = { version = "1", features = ["full", "test-util"] }

# Test utilities
pretty_assertions = "1.4"
test-case = "3.3"
```

### CI Configuration

```yaml
# .github/workflows/test.yml
name: Tests

on: [push, pull_request]

jobs:
unit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- run: cargo test --lib

integration:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- run: cargo test --test integration_*

snapshot:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- run: cargo insta test --accept
- run: git diff --exit-code

e2e:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- name: Install Playwright
run: npx playwright install chromium
- name: Build and serve
run: |
cargo build --release
./target/release/mdbook-htmx serve tests/fixtures/example-book &
sleep 5
- run: cargo test --test e2e_*

coverage:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
with:
components: llvm-tools-preview
- uses: taiki-e/install-action@cargo-llvm-cov
- run: cargo llvm-cov --all-features --lcov --output-path lcov.info
- uses: codecov/codecov-action@v3
with:
files: lcov.info
```

## Coverage Requirements

| Area | Minimum Coverage |
|------|-----------------|
| Core renderer | 80% |
| Frontmatter parsing | 90% |
| Template rendering | 70% (snapshot tests) |
| Error handling | 85% |
| Overall | 75% |

## Consequences

### Positive

- High confidence in correctness
- Fast feedback during development
- Regression prevention
- Living documentation

### Negative

- Initial test setup takes time
- Snapshot tests need maintenance
- E2E tests are slower
- Headless browser adds complexity

### Mitigation

- Use test fixtures for quick setup
- Automate snapshot updates in CI
- Run E2E tests only on PR/main
- Cache browser binaries in CI

## Alternatives Considered

### Manual Testing Only

Rely on manual QA for validation.

**Rejected** because:
- Slow and error-prone
- No regression prevention
- Doesn't scale

### Only Unit Tests

Skip integration and E2E tests.

**Rejected** because:
- Misses interaction bugs
- Can't verify HTMX behavior
- Templates need rendering tests

### Property-Based Testing Only

Use proptest/quickcheck for everything.

**Rejected** because:
- Harder to reason about failures
- Not suitable for all scenarios
- Complementary, not replacement

## References

- [Rust Testing Book](https://doc.rust-lang.org/book/ch11-00-testing.html)
- [insta Snapshot Testing](https://insta.rs/)
- [assert_cmd](https://docs.rs/assert_cmd/latest/assert_cmd/)
- [Playwright for Rust](https://github.com/nicktrav/playwright-rust)
- [Test Pyramid (Martin Fowler)](https://martinfowler.com/articles/practical-test-pyramid.html)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.