Review alternative metrics for model selection

What loss function should I use to evaluate the quality of an approximating model Q to a reference model P, when P is the distribution over emissions associated with each sector of Canada's national greenhouse gas inventory (in standard units of CO2e). The sum across sectors is important, but matching the distributions in each area is also important. I'm thinking the key evaluation metric should be a weighted sum of KL divergences per sector, with the weights being the absolute value of expected emissions from that sector. Is there a name for that type of metric? What alternative metrics might be interesting to consider?

Your intuition to weight the errors by the expected volume of emissions is exactly right for this domain. In greenhouse gas (GHG) accounting, a 5% error in a massive sector like Oil & Gas or Transportation matters far more to Canada's national climate targets than a 50% error in a small sub-sector.

Here is a breakdown of what your proposed metric is called, its pros and cons in this specific context, and some alternative metrics that might be even better suited for physical emissions data.

The Name of Your Metric

Your proposed metric is a Weighted Kullback-Leibler Divergence.

If you normalize your weights so they sum to 1—by dividing each sector's expected emissions by the total expected national emissions, $w_i = \mathbb{E}[P_i] / \sum_j \mathbb{E}[P_j]$—your metric becomes mathematically equivalent to the Conditional KL Divergence:

$$D_{KL}(P(X\vert{}S) \vert{}\vert{} Q(X\vert{}S)) = \sum_{s \in S} P(s) D_{KL}(P(X\vert{}S=s) \vert{}\vert{} Q(X\vert{}S=s))$$

Where $S$ represents the categorical distribution of sectors, weighted by their total emissions contribution.

A Caveat on Using KL Divergence Here

While KL divergence is the gold standard in information theory, it has a major drawback for physical quantities like CO2e: it is unitless (measured in nats or bits) and ignores metric distance.

If your reference model $P$ predicts emissions will be centered at 50 Megatonnes (Mt), KL divergence penalizes a $Q$ centered at 60 Mt exactly the same as it penalizes a $Q$ centered at 500 Mt, provided their density overlaps look mathematically similar. For GHG inventories, being off by 10 Mt is vastly preferable to being off by 450 Mt.

Better Alternatives for GHG Distributions

For evaluating distributions of physical, continuous variables where the "distance" between values matters, you should look toward metrics used in environmental forecasting and meteorology.

1. Continuous Ranked Probability Score (CRPS)

If you want to evaluate how well $Q$ approximates $P$, CRPS is arguably the best metric for this use case. It measures the difference between the cumulative distribution functions (CDFs) of the two models.

• Why it fits: Unlike KL divergence, the unit of CRPS is the exact same as the unit of your data (CO2e). A CRPS of 5 means your distribution is "off" by an average of 5 units of CO2e.
• How to apply it: You can take the weighted sum of the CRPS for each sector, using your absolute expected emissions weights: $\sum_i w_i \text{CRPS}(P_i, Q_i)$.

2. Wasserstein Distance (Earth Mover's Distance)

Wasserstein distance measures the minimum "work" required to transform distribution $Q$ into distribution $P$.

• Why it fits: It explicitly respects the geometry of the space. If $Q$ predicts 100 Mt of CO2e and $P$ says 110 Mt, the Wasserstein distance is small. It gracefully handles situations where $P$ and $Q$ have disjoint supports (e.g., $Q$ predicts a range of 10–20, $P$ predicts 30–40), whereas KL divergence would explode to infinity in that scenario.

3. A Composite "Marginal + Aggregate" Score

You mentioned that "the sum across sectors is important."

Weighting the sector-level errors does not guarantee that the distribution of the national total is accurate. If your approximating model $Q$ ignores correlations between sectors (e.g., if oil production goes up, electricity generation emissions might also go up), your sector-by-sector distributions might look fine, but your national total distribution $Q(\sum X_i)$ will have the wrong variance compared to $P(\sum X_i)$.

You should consider a composite metric that explicitly evaluates both the parts and the whole:

$$\mathcal{L} = \alpha D(P_{\text{total}} \vert{}\vert{} Q_{\text{total}}) + (1 - \alpha) \sum_{i} w_i D(P_i \vert{}\vert{} Q_i)$$

(Where $D$ is a distance metric like CRPS or Wasserstein, and $\alpha$ tunes the importance of the national aggregate versus sector-level accuracy).

4. The Energy Score

If you want to capture everything in one mathematical swoop, the Energy Score is the multivariate generalization of CRPS. Instead of evaluating each sector individually and summing them, the Energy Score evaluates the joint distribution of all sectors at once. This naturally forces $Q$ to learn the covariance structure between Canada's economic sectors, ensuring the sum total behaves correctly.

No files, entry points, or tests are mentioned. First clarify whether this is a request for discussion or implementation, then define the target metric, weighting, aggregate behavior, and acceptance criteria before locating the model-selection evaluation code.