easystats / easystats/performance

FR: improve formatting of multivariate models by `model_performance`

Open
#430 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Bug :bug:
Dominant language
R
Stars
1.2k
Forks
109
Avg merge
6h 34m
Merged PRs (30d)
8

Description

First, for multivariate linear models (class mlm), as currently help("sigma", package = "stats") states:

For multivariate linear models (class "mlm"), a vector of sigmas is returned, each corresponding to one column of Y.

This is the same for insight::get_sigma, having an odd output. Let's see an example (taken from https://bookdown.org/egarpor/PM-UC3M/lm-iii-mult.html):

# Dimensions and sample size
p <- 3
q <- 2
n <- 100

# A quick way of creating a non-diagonal (valid) covariance matrix for the
# errors
Sigma <- 3 * toeplitz(seq(1, 0.1, l = q)) 
set.seed(12345)
X <- mvtnorm::rmvnorm(n = n, mean = 1:p, sigma = diag(0.5, nrow = p, ncol = p))
E <- mvtnorm::rmvnorm(n = n, mean = rep(0, q), sigma = Sigma)

# Linear model
B <- matrix((-1)^(1:p) * (1:p), nrow = p, ncol = q, byrow = TRUE)
Y <- X %*% B + E

# Fitting the model (note: Y and X are matrices!)
mod <- lm(Y ~ X)
model_performance(mod)
# Indices of model performance

AIC     |     BIC | Response.Y1.R2 | Response.Y1.R2_adjusted | Response.Y1.Response | Response.Y2.R2 | Response.Y2.R2_adjusted | Response.Y2.Response |  RMSE | Sigma
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
764.558 | 777.583 |          0.703 |                   0.694 |                   Y1 |          0.787 |                   0.780 |                   Y2 | 1.599 | 1.695
764.558 | 777.583 |          0.703 |                   0.694 |                   Y1 |          0.787 |                   0.780 |                   Y2 | 1.599 | 1.567
Warning message:
In (function (..., row.names = NULL, check.rows = FALSE, check.names = TRUE,  :
  row names were found from a short variable and have been discarded

So, unique AIC, BIC and RMSE duplicated, different responses r2's in wide format duplicated again, and 2 sigma values (where should be nice to have a Sigma matrix).

In second place, for more general multiple and multivariate responses models, as those allowed by VGAM-package (see here the last point in https://github.com/easystats/insight/issues/578), so the model_performance output could be just a row with a Sigma matrix value for multivariate responses, and being so many rows as multiple responses having those models (again, as in the issue referred, cf. section 3.5.1, second half of page 115 in book doi:10.1007/978-1-4939-2818-7)
Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the multivariate lm example with model_performance and compare its duplicated rows, response-specific metrics, and Sigma values with insight::get_sigma. Read the linked insight issue 578 and the VGAM multivariate-response discussion to understand the broader cases. Done means a consistent output format for multivariate and multiple-response models, including an appropriate Sigma representation.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.