easystats / easystats/performance
FR: improve formatting of multivariate models by `model_performance`
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 1.2k
- Forks
- 109
- Avg merge
- 6h 34m
- Merged PRs (30d)
- 8
Description
First, for multivariate linear models (class mlm), as currently help("sigma", package = "stats") states:
For multivariate linear models (class "mlm"), a vector of sigmas is returned, each corresponding to one column of Y.
This is the same for insight::get_sigma, having an odd output. Let's see an example (taken from https://bookdown.org/egarpor/PM-UC3M/lm-iii-mult.html):
# Dimensions and sample size
p <- 3
q <- 2
n <- 100
# A quick way of creating a non-diagonal (valid) covariance matrix for the
# errors
Sigma <- 3 * toeplitz(seq(1, 0.1, l = q))
set.seed(12345)
X <- mvtnorm::rmvnorm(n = n, mean = 1:p, sigma = diag(0.5, nrow = p, ncol = p))
E <- mvtnorm::rmvnorm(n = n, mean = rep(0, q), sigma = Sigma)
# Linear model
B <- matrix((-1)^(1:p) * (1:p), nrow = p, ncol = q, byrow = TRUE)
Y <- X %*% B + E
# Fitting the model (note: Y and X are matrices!)
mod <- lm(Y ~ X)
model_performance(mod)
# Indices of model performance
AIC | BIC | Response.Y1.R2 | Response.Y1.R2_adjusted | Response.Y1.Response | Response.Y2.R2 | Response.Y2.R2_adjusted | Response.Y2.Response | RMSE | Sigma
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
764.558 | 777.583 | 0.703 | 0.694 | Y1 | 0.787 | 0.780 | Y2 | 1.599 | 1.695
764.558 | 777.583 | 0.703 | 0.694 | Y1 | 0.787 | 0.780 | Y2 | 1.599 | 1.567
Warning message:
In (function (..., row.names = NULL, check.rows = FALSE, check.names = TRUE, :
row names were found from a short variable and have been discarded
So, unique AIC, BIC and RMSE duplicated, different responses r2's in wide format duplicated again, and 2 sigma values (where should be nice to have a Sigma matrix).
In second place, for more general multiple and multivariate responses models, as those allowed by VGAM-package (see here the last point in https://github.com/easystats/insight/issues/578), so the model_performance output could be just a row with a Sigma matrix value for multivariate responses, and being so many rows as multiple responses having those models (again, as in the issue referred, cf. section 3.5.1, second half of page 115 in book doi:10.1007/978-1-4939-2818-7)
Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the multivariate lm example with model_performance and compare its duplicated rows, response-specific metrics, and Sigma values with insight::get_sigma. Read the linked insight issue 578 and the VGAM multivariate-response discussion to understand the broader cases. Done means a consistent output format for multivariate and multiple-response models, including an appropriate Sigma representation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100