OHDSI / OHDSI/FeatureExtraction

Tidy covariates does not remove redundant covariates if timeId is NA

Open
#137 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
74
Forks
63
PR merge metrics
No merged PRs in 30d

Description

When extracting temporal covariates either with temporalCovariateSettings or the newer temporalSequenceCovariateSettings covariates that are not temporal will get a timeId of NA. If the covariate is marked for deletion in tidyCovariates it´s supposed to be deleted by lines 107-108 in here:

 newCovariates <- newCovariates %>%
            anti_join(covariateData$deleteCovariateTimeIds, by = c("covariateId", "timeId"))

But since one of the variables to join on is NA nothing will happen. There is a field for joins "na_matches" to have NA values match each other but it seems to be only supported for dataframes, see here:

To match NA values, pass na_matches = "na" to the join verbs; this is only supported for data frames.

A small repro would be:

df <- Andromeda::andromeda(a=tibble(x=1:3, y=NA), b=tibble(x=1:1, y=NA))
df$a %>% anti_join(df$b, by=c("x", "y"))

The first line should be removed from df$a but isn't. I guess a possible workaround would be to only join on covariateId when timeId is NA, but I couldn't quickly figure out a good way to do that with an andromeda object.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in R/Normalization.R at lines 107-108 and trace tidyCovariates through the temporalCovariateSettings and temporalSequenceCovariateSettings paths. Run the Andromeda/tibble anti_join reproduction from the issue, then add a regression test covering covariateId matches where timeId is NA. Done means redundant non-temporal covariates are removed without breaking temporal joins.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.