google / google/patrick

Evaluating tests in their own frame to match test that behavior

Open
#43 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
R
Stars
145
Forks
16
PR merge metrics
No merged PRs in 30d

Description

Should patrick tests not be running in their own call stack, similar to testthat?

For various reasons I like to use defer to do cleanup after a test, for example in mocking or setting global options.

However, with Patrick it seems like the test environment is run in the global environment/call frame, so the mocks do not get cleaned up when the test is finished running (because the environment they were mocked in never exits).

Ideally of course I wouldn't have to modify the global environment in my tests, but for the time being I must, and testthat behaves as expected.
This was a particularly tricky issue to discover.

This is shown in the below reproduction, the value of `val` is restored after testthat calls, because although the environment is still the global environment, the frame number has changed (and presumably that frame ends, triggering a defer call). However, with Patrick, `val` is not cleaned up after the mock test, resulting in the second test failing because the global value has changed.

If this is intended behavior, I would appreciate a suggestion for a work around.

``` r
{
mock_value <- function(name, replacement, parent = 1) {
old_value <- get(name, envir = .GlobalEnv)
assign(name, replacement, envir = .GlobalEnv)
withr::defer({
assign(name, old_value, envir = .GlobalEnv)
print("ran defer block")
}, envir = parent.frame(parent))
}
val = TRUE
testthat::test_that("", {
mock_value("val", FALSE)
print("initial test that mock")
print(val)
print(sys.frame())
print(sys.nframe())
testthat::expect_false(val)
})
print(val)
testthat::test_that("", {
print("second test that no mock")
print(val)
print(sys.frame())
print(sys.nframe())
testthat::expect_true(val)
})
print(val)
patrick::with_parameters_test_that(":", .cases = list(a = ""),
{
mock_value("val", FALSE)
print("initial with parameters test that mock")
print(val)
print(sys.frame())
print(sys.nframe())
testthat::expect_false(val)
})
print(val)
patrick::with_parameters_test_that(":", .cases = list(a = ""),
{
print("second with parameters test that no mock")
print(val)
print(sys.frame())
print(sys.nframe())
testthat::expect_true(val)
})
print(val)
}
#> [1] "initial test that mock"
#> [1] FALSE
#>
#> [1] 52
#> [1] "ran defer block"
#> Test passed with 1 success 🌈.
#> [1] TRUE
#> [1] "second test that no mock"
#> [1] TRUE
#>
#> [1] 52
#> Test passed with 1 success πŸŽ‰.
#> [1] TRUE
#> [1] "initial with parameters test that mock"
#> [1] FALSE
#>
#> [1] 0
#> Test passed with 1 success πŸ₯‡.
#> [1] FALSE
#> [1] "second with parameters test that no mock"
#> [1] FALSE
#>
#> [1] 0
#> ── Failure: : a= ───────────────────────────────────────────────────────────────
#> Expected `val` to be TRUE.
#> Differences:
#> `actual`: FALSE
#> `expected`: TRUE
#>
#> Backtrace:
#> β–†
#> 1. β”œβ”€rlang::eval_tidy(code, test_args)
#> 2. └─testthat::expect_true(val)
#> Error in `pmap()`:
#> β„Ή In index: 1.
#> Caused by error:
#> ! Test failed with 1 failure and 0 successes.
```

Created on 2026-05-08 with [reprex v2.1.1](https://reprex.tidyverse.org)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.