BlueBrain / BlueBrain/morphoclass

`WARNING: The least populated class in y has only 2 members, which is less than n_splits=3`

Open
#22 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
12
Forks
6
PR merge metrics
No merged PRs in 30d

Description

## Context & Description

When we run k-fold cross validation, we use `n_splits=3`.
https://github.com/BlueBrain/morphoclass/blob/021c632d2214ab25b5e4f58827f85420d536b66e/dvc/training/configs/splitter-stratifKFold.yaml#L4

But for layer `L4` and layer `L6` of the dataset `interneurons`, we have classes `L4_BP` and `L6_DBC` with `2<3` samples.
Counts per Interneurons subclass


This situation generates the following Python warning when iterating over `StratifiedKFold.split(X, y)`:
```Python
UserWarning: The least populated class in y has only 2 members, which is less than n_splits=3.
```

## What happens?

If a class has less members than `n_splits`, then for some splits of `StratifiedKFold` we will have no representatives of that class in the validation or in the training set. For instance, splitting `[0] * 7 + [1] * 3` with `StratifiedKFold(n_splits=3)` yields the following training and validation sets, where in `split 3` there is no sample of class `1` in the validation set!
```sh
train-set --- valid-set
[1, 0, 0, 0, 0, 0] --- [1, 0, 0, 0] # split 1
[1, 0, 0, 0, 0, 0, 0] --- [1, 0, 0] # split 2
[1, 1, 0, 0, 0, 0, 0] --- [0, 0, 0] # split 3
```

## Why may this be an issue?
1. **Some metrics may not be computed and/or return wrong values.**
Metrics such as `precision_score`, `recall_score`, `f1_score` cannot be computed if there is no sample for a given class.
For instance, in the example above, using the validation set of `split 3` to compute `f1_score` for `y_pred = [0, 0, 0]` will return `0.0` (despite `y_pred` matching perfectly `y_true`!!) and raise this warning:
```python
UndefinedMetricWarning:
Precision is ill-defined and being set to 0.0 due to no predicted samples.
Use `zero_division` parameter to control this behavior.
```
**However** in our case we do not compute metrics per-split and then average across all splits, but instead we take all the out-of-sample predictions (generated during the various splits) and then compute the metric using all samples. Therefore, no class can ever have `0` samples during evaluation.

2. **A class in the validation set may not be present in the training set.**
This would be dramatic, because after the training the model would not even be aware the existence of a class that is however present in the validation set. So it is guaranteed that the model will never predict that class.

**However** I have never observed this happening on our data. I am not even sure it is possible.

3. **Evaluating on classes with few samples may be not very meaningful.**
Does it really make sense to take into account the model performance with respect to a class that has only 1 member in the training set or in the validation set?

**However** as long as we look at `micro` or `weighted` averages, the impact of (potentially awful) performance on classes with 1 or 2 samples is limited. But this could be bad if we want to look at `macro` averages. https://github.com/scikit-learn/scikit-learn/blob/baf828ca126bcb2c0ad813226963621cafe38adb/sklearn/metrics/_classification.py#L1049-L1062

## How do we solve this?

We could remove or merge classes with less than 3 samples.
But this should be discussed with the scientists. Maybe those classes with few samples are very important and well defined and must be kept anyway?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.