uber / uber/causalml

mape silently returns NaN when all targets are near-zero (empty filtered slice)

Open
#1,047 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
6k
Forks
877
PR merge metrics
No merged PRs in 30d

Description

Summary

mape silently returns NaN when every target is (near-)zero. The near-zero filter then selects nothing, and np.mean of an empty slice yields NaN (plus a RuntimeWarning) instead of a clear error — while the scalar sibling ape asserts against exactly this case.

Location

  • File: causalml/metrics/regression.py
  • Functions: mape(y, p) (primary); smape(y, p) has the analogous all-y == p == 00/0 → NaN path
  • Constant: causalml/metrics/const.py, EPS = 1e-15

Relevant code path (static analysis of current main):

def mape(y, p):
    filt = np.abs(y) > EPS
    return np.mean(np.abs(1 - p[filt] / y[filt]))  # mean of empty slice -> nan when filt is all False
def ape(y, p):
    assert np.abs(y) > EPS  # scalar version guards; vector version does not
    return np.abs(1 - p / y)

Problem

All-zero (or all-sub-EPS) target vectors are realistic in uplift work — e.g. scoring a slice with zero conversions. mape then computes np.mean over an empty selection and returns nan, which propagates silently through model comparisons and leaderboards. The only signal is a numpy RuntimeWarning ("Mean of empty slice"), easily lost in training logs. smape shares the shape of the problem: with y == p == 0 everywhere, the ratio is 0/0 → NaN per element, and the mean is NaN.

Trigger / Reproduction

Based on static analysis (no execution performed):

import numpy as np
from causalml.metrics import mape
mape(np.zeros(100), np.zeros(100))  # filt all False -> np.mean([]) -> nan

Note: this is a static-analysis finding; I did not run the snippet, but the numpy empty-slice-means-nan semantics are documented behavior.

Expected Behavior

An explicit, actionable failure (e.g. ValueError when no target exceeds EPS) — or at minimum a causalml logger warning naming the function — consistent with ape's refusal to divide by ~zero. Never a silent NaN metric.

Actual Behavior

Returns NaN with only a numpy RuntimeWarning, indistinguishable downstream from a computed score.

Impact

  • Silent leaderboard corruption: NaN sorts/aggregates unpredictably, and a "best model" pick can hinge on it.
  • Inconsistency within the same module: scalar ape guards, vector mape/smape do not.

Suggested Direction

  • Guard mape (and the smape all-zero case) with an explicit check on the filtered selection, raising ValueError or logging a warning and returning NaN deliberately — maintainer's choice, but make it explicit and documented in the docstrings.

Evidence

  • Source via API: regression.py mape/smape/ape as quoted; const.py EPS.
  • Duplicate check: issue searches for mape NaN empty (total_count: 0) and metric nan zero (only unrelated closed feature requests #994, #887) — no apparent duplicate.

Classification

  • FACT: all-~zero y makes mape take the mean of an empty slice (verified in source via API).
  • INFERENCE: the result is NaN propagated as if it were a score.
  • HYPOTHESIS: an explicit guard preserves all current behavior on valid inputs while making the degenerate case visible.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in causalml/metrics/regression.py by reading mape, smape, and ape, then check EPS in causalml/metrics/const.py and run the supplied zero-target reproduction. Add focused coverage for the degenerate cases and make the failure or warning explicit and documented, without changing valid-input behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.