insightsengineering / insightsengineering/teal.picks
[Feature Request]: Build output table with join_keys to merged datasets and primary keys
- Dominant language
- R
- Stars
- 4
- Forks
- 0
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 3
Description
### Feature description
wip on the test case
Currently, the `join_keys` is not being generated for `ANL` and is a limitation for future merges being done in the modules.
It was on the original design of `teal.picks` with this comment that still lives in the code:
```r
# glossary:
# dataset/dataname: dataset (or its name) in the current iteration (datasets are merged in a loop)
# anl datasets/datanames: datasets (or names) which anl is composed of (this is a cumulative process)
# remaining datasets/datanames: datasets (or names) which are about to be merged
#
# Rules:
# 1. anl "inherits" foreign keys from anl datasets to remaining datasets
# 2. foreign keys of current dataset are added to anl join_keys but only if no relation from anl already.
# 3. foreign keys should be renamed if duplicated with anl colnames
# 4. (for later) selected datasets might not be directly mergable, we need to find the "path" which
# will probably involve add intermediate datasets in between to perform merge
# 5. selected variables are added to anl.
# 6. duplicated variables added to anl should be renamed
# remaining_datanames <- setdiff(remaining_datanames, dataname)
```
#### Reproducible example
```r
pkgload::load_all("../.teal.picks")
data <- within(teal.data::teal_data(), {
ADSL <- teal.data::rADSL
ADLB <- teal.data::rADLB
ADLB$AGE <- NULL
ADLB$AGEU <- NULL
})
teal.data::join_keys(data) <- teal.data::default_cdisc_join_keys[names(data)]
shiny::testServer(
merge_srv,
expr = {
print("Output table")
print(session$returned$data()$ANL)
print("")
print("--")
print("Join keys")
print(teal.data::join_keys(session$returned$data()))
},
args = list(
id = "test",
data = shiny::reactive(data),
output_name = "ANL",
selectors = list(
a = shiny::reactive(picks(
datasets(choices = "ADLB", selected = "ADLB"),
variables(choices = colnames(data$ADLB), selected = c("AVAL"))
))
)
)
)
#> [1] "Output table"
#> # A tibble: 8,400 × 3
#> STUDYID USUBJID AVAL
#>
#> 1 AB12345 AB12345-BRA-1-id-105 4.30
#> 2 AB12345 AB12345-BRA-1-id-105 24.7
#> 3 AB12345 AB12345-BRA-1-id-105 24.9
#> 4 AB12345 AB12345-BRA-1-id-105 3.67
#> 5 AB12345 AB12345-BRA-1-id-105 18.5
#> 6 AB12345 AB12345-BRA-1-id-105 7.73
#> 7 AB12345 AB12345-BRA-1-id-105 23.5
#> 8 AB12345 AB12345-BRA-1-id-105 9.09
#> 9 AB12345 AB12345-BRA-1-id-105 9.15
#> 10 AB12345 AB12345-BRA-1-id-105 10.9
#> # ℹ 8,390 more rows
#> # ℹ Use `print(n = ...)` to see more rows
#> [1] "Join keys"
#> A join_keys object containing foreign keys between 2 datasets:
#> ADSL: [STUDYID, USUBJID]
#> <-- ADLB: [STUDYID, USUBJID]
#> ADLB: [STUDYID, USUBJID, PARAMCD, AVISIT]
#> --> ADSL: [STUDYID, USUBJID]
```
#### Test case
```r
# test file
pkgload::load_all("/home/averissimo/work/roche🟦/packages📦/teal.picks")
describe("Output table builds valid join_keys", {
data <- within(teal.data::teal_data(), {
ADSL <- teal.data::rADSL
ADLB <- teal.data::rADLB
ADLB$AGE <- NULL
ADLB$AGEU <- NULL
})
teal.data::join_keys(data) <- teal.data::default_cdisc_join_keys[names(data)]
it("Single merge produces join_keys identical to ADLB ", {
shiny::testServer(
merge_srv,
expr = {
expect_mapequal(
teal.data::join_keys(session$returned$data())["ADLB", "ADLB"],
teal.data::join_keys(session$returned$data())["ANL", "ANL"]
)
},
args = list(
id = "test",
data = shiny::reactive(data),
output_name = "ANL",
selectors = list(
a = shiny::reactive(picks(
datasets(choices = "ADLB", selected = "ADLB"),
variables(choices = colnames(data$ADLB), selected = c("AVAL"))
))
)
)
)
})
it("merge with 2 datasets produces valid join_keys ", {
shiny::testServer(
merge_srv,
expr = {
expect_setequal(
union(
teal.data::join_keys(session$returned$data())["ADLB", "ADLB"],
teal.data::join_keys(session$returned$data())["ADSL", "ADSL"]
),
unname(teal.data::join_keys(session$returned$data())["ANL", "ANL"])
)
},
args = list(
id = "test",
data = shiny::reactive(data),
output_name = "ANL",
selectors = list(
a = shiny::reactive(picks(
datasets(choices = "ADLB", selected = "ADLB"),
variables(choices = colnames(data$ADLB), selected = c("AVAL"))
)),
b = shiny::reactive(picks(
datasets(choices = "ADSL", selected = "ADSL"),
variables(choices = colnames(data$ADSL), selected = c("AGE"))
))
)
)
)
})
it("merge with modified join_keys (complex join_keys that do not match names) produces valid join_keys ", {
data <- within(teal.data::teal_data(), {
ADSL <- teal.data::rADSL
ADLB <- teal.data::rADLB
ADLB$USUBJID_2 <- ADLB$USUBJID
ADLB$USUBJID <- 1
})
teal.data::join_keys(data) <- teal.data::default_cdisc_join_keys[names(data)]
teal.data::join_keys(data)["ADLB", "ADLB"] <- c("STUDYID", "USUBJID", "USUBJID_2")
teal.data::join_keys(data)["ADSL", "ADLB"] <- c("STUDYID", "USUBJID" = "USUBJID_2")
shiny::testServer(
merge_srv,
expr = {
print(teal.data::join_keys(session$returned$data()))
print(head(session$returned$data()$ANL))
expect_setequal(
colnames(session$returned$data()$ANL),
c("STUDYID", "USUBJID", "USUBJID_2")
)
expect_setequal(
union(
teal.data::join_keys(session$returned$data())["ADLB", "ADLB"],
teal.data::join_keys(session$returned$data())["ADSL", "ADSL"]
),
unname(teal.data::join_keys(session$returned$data())["ANL", "ANL"])
)
},
args = list(
id = "test",
data = shiny::reactive(data),
output_name = "ANL",
selectors = list(
a = shiny::reactive(picks(
datasets(choices = "ADLB", selected = "ADLB"),
variables(choices = colnames(data$ADLB), selected = c("AVAL"))
)),
b = shiny::reactive(picks(
datasets(choices = "ADSL", selected = "ADSL"),
variables(choices = colnames(data$ADSL), selected = c("AGE"))
))
)
)
)
})
})
```
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct.
### Contribution Guidelines
- [x] I agree to follow this project's Contribution Guidelines.
### Security Policy
- [x] I agree to follow this project's Security Policy.
Contributor guide
Research direction
Start with the merge_srv entry point and the supplied test case, focusing on how teal.data::join_keys is carried into the ANL output. Run the single-dataset, two-dataset, and modified-key scenarios shown in the issue. Done means ANL exposes valid join_keys for each selected dataset, including renamed or non-matching key columns, with the expectations in the test case passing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100