insightsengineering / insightsengineering/teal.code
[Discussion]: Handling operations that use/modify RNG
- Dominant language
- R
- Stars
- 12
- Forks
- 11
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 1
Description
### What is your question?
Per discussion with @llrs-roche, here are some topics for the contributors to discuss and research.
### Problem
Random/stoichastic operations are not being well handled in `within`/`teal.code::eval_code`, in particular when code parse is being used.
In the section below you can observe that `q |> get_code(names = "IRIS")` is not fully reproducible as the random operations before were not included, nor was the RNG state updated before code execution
### Reproducible code
This code highlights the 2 main problems:
1. `set.seed()` and other random operations are not kept in partial code (`q |> get_code(names = "XXX")`)
1. `MTCARS`/`IRIS` random operations are influenced by preceding code
```r
library(teal.code)
q <- qenv() |>
within({
IRIS <- datasets::iris
MTCARS <- datasets::mtcars
set.seed(1)
.random <- runif(1)
IRIS$new <- runif(nrow(IRIS))
MTCARS$new <- runif(nrow(MTCARS))
})
q |> get_code() |> cat()
#> iris <- datasets::iris
#> mtcars <- datasets::mtcars
#> set.seed(1)
#> .random <- runif(1)
#> iris$new <- runif(nrow(iris))
#> mtcars$new <- runif(nrow(mtcars))
q |> get_code(names = "IRIS") |> cat()
#> IRIS <- datasets::iris
#> IRIS$new <- runif(nrow(IRIS))
local({
eval(str2expression(q |> get_code(names = "IRIS")))
q$IRIS |> rlang::hash() |> cat("# hash qenv IRIS\n")
IRIS |> rlang::hash() |> cat("# hash reproduced IRIS\n")
})
#> 9bba33c8ed5cd90cd13316f35f812f55 # hash qenv IRIS
#> c90a89c1814206bc19fdc32d04d1eefa # hash reproduced IRIS
q |> get_code(names = "MTCARS") |> cat()
#> MTCARS <- datasets::mtcars
#> MTCARS$new <- runif(nrow(MTCARS))
local({
eval(str2expression(q |> get_code(names = "MTCARS")))
q$mtcars |> rlang::hash() |> cat("# hash qenv MTCARS\n")
mtcars |> rlang::hash() |> cat("# hash reproduced MTCARS\n")
})
#> 2c0a8a99dc147d5445c3b49d035665b2 # hash qenv MTCARS
#> 6755d143ff87b73a1196c186cec7e86a # hash reproduced MTCARS
```
### Reference material
- ref1: https://github.com/insightsengineering/teal.code/issues/278
- ref2: https://www.jottr.org/2020/09/21/detect-when-the-random-number-generator-was-used/
- Detect when random seed is modified
- ref3: https://blog.djnavarro.net/posts/2023-12-27_seedcatcher/
### Possible set of solutions
- Use `teal.code::eval_code()` and `# @linksto XXXXX` to include random operations
- Status quo
- Cons: Requires character base code execution and requires manual maintenance
- Detect random seed at start and inform user
- Track random seed and incorporate it in code execution
- When using code parser, re-run partial code to sync modules with reproducible code (while keeping `set.seed()` expressions)
- cons: slow down teal app even more and data is not consistent with initial `data` argument in teal.
- Always include `set.seed()` expression and hope it doesn't have a big impact
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct.
### Contribution Guidelines
- [x] I agree to follow this project's Contribution Guidelines.
### Security Policy
- [x] I agree to follow this project's Security Policy.
Contributor guide
Research direction
Start with the within and teal.code::eval_code code paths, then trace how get_code(names = ...) and the code parser construct partial code. Read issue 278 and the linked RNG references to understand seed tracking and reproducibility. Done should mean the contributors agree on and implement a reproducible policy for random operations in partial-code execution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100