Hugely different R2 values for slightly different models

I've been fitting some mixed models to genomic (transcriptomic) count data, and wanted to assess model fit. I've tried out a few models and a few options for R2, however they give wildly different R2 values.

I have a couple of questions:

• Could this be a bug? I kind of expect not, but worth checking.
• Are there reasons why this data is not appropriate for the Nakagawa R2 calculation? If so, are there general principles that could be added to the documentation?
• Are these different ways of calculating R2 just not comparable in this way? If so, hopefully this could be a useful case study.

Thanks!
Will

MWE

The data has 8 different celltypes, across 196 individuals (for almost all individuals we have all 8 possible values). counts corresponds to how many times we observe a given gene, libsize is the offset for this observation, and logcpm = log( counts / libsize * 1e6 + 1) i.e. log values slightly pushed away from 0 and normalized to the expected range.

Plotting these log values indicates that we expect both celltype and invididual to be relevant:

I fit 4 models and calculate R2 for each:

## model 1: negative binomial with random effect
fit_glmm    = glmmTMB::glmmTMB(counts ~ celltype + (1|individual),
  offset = log(libsize), data = data_dt, family = glmmTMB::nbinom2)
performance::r2(fit_glmm)
# # R2 for Mixed Models
#   Conditional R2: 0.084
#      Marginal R2: 0.054
# 4: mu of 0.0 is too close to zero, estimate of random effect variances may be
#   unreliable.

## model 2: negative binomial without random effect
fit_glm     = glm(counts ~ celltype,
  offset = log(libsize), data = data_dt, family = glmmTMB::nbinom2)
performance::r2(fit_glm)
# # R2 for Generalized Linear Regression
#   Nagelkerke's R2: 0.738

## model 3: log CPM with random effect
fit_lmer    = lme4::lmer(logcpm ~ celltype + (1 | individual), data = data_dt)
performance::r2(fit_lmer)
# # R2 for Mixed Models

#   Conditional R2: 0.444
#      Marginal R2: 0.215

## model 4: log CPM without random effect
fit_lm      = lm(logcpm ~ celltype, data = data_dt)
performance::r2(fit_lm)
# # R2 for Linear Regression
#        R2: 0.214
#   adj. R2: 0.211

The main thing I don't understand is why there is such a large difference between models 1 and 2. An R2 of 8% just doesn't seem plausible when the distributions are so tight (while an R2 of 74% for model 2 seems about right). The coefficient estimates are pretty similar, so it seems more likely that the difference comes from how the R2 is calculated, rather the fits being wildly different.

Does the warning for model 1 indicate that there are issues here? If so, what could they be? To me it doesn't seem like an unreasonably complex model for the data...

I've half-read the Nakagawa paper, so could it be that for model 1 the R2 is based on log values, while for R2 it uses deviance? Could that explain the huge difference?

(The differences between the GLM and log-linear models seem to make sense, as with many zeros present the GLM should better capture the data.)

Start by reproducing the four-model MWE in the issue and reading the performance::r2 entry point for GLM and mixed-model calculations. Compare the reported R2 definitions and the warning for the glmmTMB model. Done means explaining the discrepancy and, if appropriate, documenting the relevant limitations or comparability guidance.