insightsengineering / insightsengineering/teal.data
Consider migration to `dm`
- Dominant language
- R
- Stars
- 11
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
Blocked by https://github.com/insightsengineering/teal/issues/628 - we need more insights about how chevron and teal fits together.
- Research CDISCData, CDISCFilteredData to store `dm` object to manage relationships between data.frames object.
- Research how MAE can be stored
- Can `dm` be primary data object passed to the teal_module?
------
### Some initial research
1. Easy initialization of the relational data
```r
library(dm)
ADSL <- synthetic_cdisc_data("latest")$adsl
ADTTE <- synthetic_cdisc_data("latest")$adtte
ADRS <- synthetic_cdisc_data("latest")$adrs
IRIS <- iris
IRIS$id <- seq_len(nrow(IRIS))
dm <- dm(ADSL, ADTTE, ADRS, IRIS) |>
dm_add_pk(ADSL, teal::get_cdisc_keys("ADSL")) |>
dm_add_pk(ADTTE, teal::get_cdisc_keys("ADTTE")) |>
dm_add_pk(ADRS, teal::get_cdisc_keys("ADRS")) |>
dm_add_pk(IRIS, "id") |>
dm_add_fk(table = ADTTE, columns = teal::get_cdisc_keys("ADSL"), ref_table = ADSL) |>
dm_add_fk(table = ADRS, columns = teal::get_cdisc_keys("ADSL"), ref_table = ADSL)
```
2. Get keys
```r
# get primary and foreign keys
dm |> dm_get_all_pks()
dm |> dm_get_all_fks()
```
3. Check metadata
`dm` throws an error if the primary keys are duplicated
```r
# check if keys are unique
dm |> dm_examine_constraints()
dm(ADSL, ADTTE, ADRS, IRIS) |>
dm_add_pk(ADSL, teal::get_cdisc_keys("ADSL")) |>
dm_add_pk(ADTTE, teal::get_cdisc_keys("ADSL")) |>
dm_examine_constraints()
```
4. `dm` holds data in list, so the data is copied by reference
```r
# same objects
identical(dm[["ADSL"]], ADSL)
```
5. Filters can be added to metadata with the `dm_filter`
```r
# apply filter
dm |>
dm_filter(ADSL, SEX == "M") |>
dm_filter(ADRS, PARAMCD %in% c("BESRSPI", "INVET")) |>
dm_filter(ADRS, AVISIT %in% c("BASELINE", "SCREENING", "CYCLE2", "DAY1"))
```
```r
# filter and select
dm |>
dm_filter(ADSL, SEX == "M") |>
dm_filter(ADRS, PARAMCD %in% c("BESRSPI", "INVET")) |>
dm_apply_filters() |>
dm_select(ADSL, USUBJID, STUDYID, AGE) |>
dm_flatten_to_tbl(ADRS) |>
nrow()
```
6. Joining is performed automatically by `dm`
```r
# join tables
joined <- dm |>
dm_filter(ADSL, SEX == "M") |>
dm_filter(ADRS, PARAMCD %in% c("BESRSPI", "INVET")) |>
dm_apply_filters() |>
dm_select(ADSL, USUBJID, STUDYID, AGE) |>
dm_select(ADRS, USUBJID, STUDYID, PARAMCD, AVISIT, AGE, SEX, AVAL) |>
dm_join_to_tbl(ADSL, ADRS, join = left_join)
```
7. `dm` object is mutable
```r
library(dm)
library(scda)
ADSL <- synthetic_cdisc_data("latest")$adsl
ADTTE <- synthetic_cdisc_data("latest")$adtte
ADRS <- synthetic_cdisc_data("latest")$adrs
IRIS <- iris
IRIS$id <- seq_len(nrow(IRIS))
dm <- dm(ADSL, ADTTE, ADRS, IRIS)
dm2 <- dm |>
dm_add_pk(ADSL, teal.data::get_cdisc_keys("ADSL")) |>
dm_add_pk(ADTTE, teal.data::get_cdisc_keys("ADTTE")) |>
dm_add_pk(ADRS, teal.data::get_cdisc_keys("ADRS")) |>
dm_add_pk(IRIS, "id") |>
dm_add_fk(table = ADTTE, columns = teal.data::get_cdisc_keys("ADSL"), ref_table = ADSL) |>
dm_add_fk(table = ADRS, columns = teal.data::get_cdisc_keys("ADSL"), ref_table = ADSL)
lobstr::obj_addr(dm)
lobstr::obj_addr(dm2)
```
8. `dm` operations preserve attributes
```r
dm_filtered <- dm |>
dm_filter(ADSL, SEX == "M") |>
dm_filter(ADRS, PARAMCD %in% c("BESRSPI", "INVET")) |>
dm_apply_filters() |>
dm_select(ADSL, USUBJID, STUDYID, AGE) |>
dm_select(ADRS, USUBJID, STUDYID, PARAMCD, AVISIT, AGE, SEX, AVAL) |>
dm_select_tbl(ADSL, ADRS) |>
dm_flatten_to_tbl(ADRS, ADSL)
attr(joined, "label")
formatters::var_labels(joined)
```
Issues:
- `dm` renames duplicated the columns automatically so we might have a problem with difference between input column names and output column names. `input$select_from_adsl` and `input$select_from_adrs` won't be found in `colnames(data)`
- ~won't support same dataset selectors by default. Following will not possible~
Not needed since we are going to fix data_merge https://github.com/insightsengineering/NEST-roadmap/issues/36
```r
ANL1 <- ADRS %>% filter(PARAMCD == "PARAM1") %>% select(, PARAM1 = AVAL)
ANL2 <- ADRS %>% filter(PARAMCD == "PARAM2") %>% select(, PARAM2 = AVAL)
ANL <- merge(ANL1, ANL2, by = c())
```
Maybe we should reconsider this problematic option and just to do this:
```r
ANL <- ADRS %>% filter(PARAMCD %in% c("PARAM1", "PARAM2")_) %>% select(, AVAL)
```
Contributor guide
Assessment
This issue has not been assessed yet.