Multiple `ZeroDivisionError` bugs in `lm_eval/api/metrics.py` aggregation functions

Multiple ZeroDivisionError bugs in lm_eval/api/metrics.py aggregation functions

Description

Several aggregation functions in lm_eval/api/metrics.py lack guards against edge-case inputs, causing ZeroDivisionError crashes that halt the entire evaluation pipeline. Three related bugs are reported here:

---

1. mean() — empty result list

@register_aggregation("mean")
def mean(arr):
    return sum(arr) / len(arr)  # ZeroDivisionError if arr is []

When a task produces zero evaluation samples (e.g., due to filtering, an empty dataset split, or a misconfigured evaluation), len(arr) is 0.

---

2. mean_stderr() / sample_stddev() — single-element or empty list

def sample_stddev(arr):
    mu = mean(arr)
    return math.sqrt(sum([(x - mu) ** 2 for x in arr]) / (len(arr) - 1))
    #                                                      ^^^^^^^^^^^
    #                                    ZeroDivisionError when len(arr) == 1

def mean_stderr(arr):
    return sample_stddev(arr) / math.sqrt(len(arr))
    #                                     ^^^^^^^^
    #                       ZeroDivisionError when len(arr) == 0

A single-sample evaluation task (valid for quick sanity checks) crashes the entire results computation.

---

3. weighted_mean() / bits_per_byte() — weights summing to zero

def weighted_mean(items):
    a, b = zip(*items)
    if len(a) == 0:       # guards empty list...
        return np.nan
    return sum(a) / sum(b)  # ...but not sum(b) == 0

The len(a) == 0 guard catches empty inputs, but if the list is non-empty and all weights (b values) are zero (zero-length documents or degenerate tokenization), sum(b) is 0 and the division still fails. This is reachable through bits_per_byte():

@register_aggregation("bits_per_byte")
def bits_per_byte(items):
    return -weighted_mean(items) / math.log(2)

---

Expected behavior

These functions should return np.nan (or 0.0) gracefully on degenerate inputs rather than crashing the pipeline.

Suggested fix

@register_aggregation("mean")
def mean(arr):
    if len(arr) == 0:
        return np.nan
    return sum(arr) / len(arr)

def sample_stddev(arr):
    if len(arr) < 2:
        return float('nan')
    mu = mean(arr)
    return math.sqrt(sum([(x - mu) ** 2 for x in arr]) / (len(arr) - 1))

def mean_stderr(arr):
    if len(arr) == 0:
        return float('nan')
    return sample_stddev(arr) / math.sqrt(len(arr))

def weighted_mean(items):
    a, b = zip(*items)
    if len(a) == 0:
        return np.nan
    total_weight = sum(b)
    if total_weight == 0:
        return np.nan
    return sum(a) / total_weight

How this was found

These were identified via static analysis, which flagged all unguarded divisions as DSE-confirmed reachable DIV_ZERO bugs.

Read lm_eval/api/metrics.py, starting with mean(), sample_stddev(), mean_stderr(), weighted_mean(), and bits_per_byte(). Trace how empty, single-element, and zero-weight inputs reach these registered aggregations. Done means degenerate inputs return a non-crashing NaN or zero result as specified, while normal inputs retain their existing behavior.