insightsengineering / insightsengineering/teal.code

[Discussion]: Handling operations that use/modify RNG

Aperta
#291 2 commenti 1 reazione 0 assegnatari Vedi su GitHub
discussion
Lingua principale
R
Stelle
12
Fork
11
Merge medio
1g 17h
PR unite (30g)
1

Descrizione

### What is your question?

Per discussion with @llrs-roche, here are some topics for the contributors to discuss and research.

### Problem

Random/stoichastic operations are not being well handled in `within`/`teal.code::eval_code`, in particular when code parse is being used.

In the section below you can observe that `q |> get_code(names = "IRIS")` is not fully reproducible as the random operations before were not included, nor was the RNG state updated before code execution

### Reproducible code

This code highlights the 2 main problems:
1. `set.seed()` and other random operations are not kept in partial code (`q |> get_code(names = "XXX")`)
1. `MTCARS`/`IRIS` random operations are influenced by preceding code

```r
library(teal.code)
q <- qenv() |>
within({
IRIS <- datasets::iris
MTCARS <- datasets::mtcars
set.seed(1)
.random <- runif(1)
IRIS$new <- runif(nrow(IRIS))
MTCARS$new <- runif(nrow(MTCARS))
})

q |> get_code() |> cat()
#> iris <- datasets::iris
#> mtcars <- datasets::mtcars
#> set.seed(1)
#> .random <- runif(1)
#> iris$new <- runif(nrow(iris))
#> mtcars$new <- runif(nrow(mtcars))

q |> get_code(names = "IRIS") |> cat()
#> IRIS <- datasets::iris
#> IRIS$new <- runif(nrow(IRIS))

local({
eval(str2expression(q |> get_code(names = "IRIS")))
q$IRIS |> rlang::hash() |> cat("# hash qenv IRIS\n")
IRIS |> rlang::hash() |> cat("# hash reproduced IRIS\n")
})
#> 9bba33c8ed5cd90cd13316f35f812f55 # hash qenv IRIS
#> c90a89c1814206bc19fdc32d04d1eefa # hash reproduced IRIS

q |> get_code(names = "MTCARS") |> cat()
#> MTCARS <- datasets::mtcars
#> MTCARS$new <- runif(nrow(MTCARS))

local({
eval(str2expression(q |> get_code(names = "MTCARS")))
q$mtcars |> rlang::hash() |> cat("# hash qenv MTCARS\n")
mtcars |> rlang::hash() |> cat("# hash reproduced MTCARS\n")
})
#> 2c0a8a99dc147d5445c3b49d035665b2 # hash qenv MTCARS
#> 6755d143ff87b73a1196c186cec7e86a # hash reproduced MTCARS
```

### Reference material

- ref1: https://github.com/insightsengineering/teal.code/issues/278
- ref2: https://www.jottr.org/2020/09/21/detect-when-the-random-number-generator-was-used/
- Detect when random seed is modified
- ref3: https://blog.djnavarro.net/posts/2023-12-27_seedcatcher/

### Possible set of solutions

- Use `teal.code::eval_code()` and `# @linksto XXXXX` to include random operations
- Status quo
- Cons: Requires character base code execution and requires manual maintenance
- Detect random seed at start and inform user
- Track random seed and incorporate it in code execution
- When using code parser, re-run partial code to sync modules with reproducible code (while keeping `set.seed()` expressions)
- cons: slow down teal app even more and data is not consistent with initial `data` argument in teal.
- Always include `set.seed()` expression and hope it doesn't have a big impact

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct.

### Contribution Guidelines

- [x] I agree to follow this project's Contribution Guidelines.

### Security Policy

- [x] I agree to follow this project's Security Policy.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Start with the within and teal.code::eval_code code paths, then trace how get_code(names = ...) and the code parser construct partial code. Read issue 278 and the linked RNG references to understand seed tracking and reproducibility. Done should mean the contributors agree on and implement a reproducible policy for random operations in partial-code execution.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
r
Ambito
backend
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Attiva
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.