Lightning-AI / Lightning-AI/torchmetrics

Add `DepthScore`

Open
#854 10 comments 0 reactions 1 assignee View on GitHub

@Sohaib-Ahmed21 is already working on this.

Since Jan 20, 2026.

enhancement New metric topic: Text
Dominant language
Python
Stars
2.5k
Forks
526
Avg merge
6d 11h
Merged PRs (30d)
5

Description

## 🚀 Feature

Add `DepthScore`

### Sources:
- Paper: [A Pseudo-Metric between Probability Distributions based on Depth-Trimmed Regions
](https://arxiv.org/pdf/2103.12711.pdf)
- [Repo](https://github.com/PierreColombo/nlg_eval_via_simi_measures/)

### Motivation

The recent NLG metrics are more often based on BERT (or related) embeddings. As such, I believe, we should also start adding such metrics into `TorchMetrics` with an extra dependency on `transformers` if a user wants to use any of these metrics. The `DepthScore` metric is from a family of untrained metrics (i.e. the model is not fine-tuned on any specific task) so it should be easier for us to begin with it.

### Abstract:
The design of a metric between probability distributions is a longstanding problem motivated by numerous applications in Machine Learning. Focusing on continuous probability distributions on the Euclidean space ℝ^d, we introduce a novel pseudo-metric between probability distributions by leveraging the extension of univariate quantiles to multivariate spaces. Data depth is a nonparametric statistical tool that measures the centrality of any element x∈ℝ^d with respect to (w.r.t.) a probability distribution or a data set. It is a natural median-oriented extension of the cumulative distribution function (cdf) to the multivariate case. Thus, its upper-level sets -- the depth-trimmed regions -- give rise to a definition of multivariate quantiles. The new pseudo-metric relies on the average of the Hausdorff distance between the depth-based quantile regions w.r.t. each distribution. Its good behavior w.r.t. major transformation groups, as well as its ability to factor out translations, are depicted. Robustness, an appealing feature of this pseudo-metric, is studied through the finite sample breakdown point. Moreover, we propose an efficient approximation method with linear time complexity w.r.t. the size of the data set and its dimension. The quality of this approximation as well as the performance of the proposed approach are illustrated in numerical experiments.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.