mape silently returns NaN when all targets are near-zero (empty filtered slice)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6k
- Forks
- 877
- PR merge metrics
- No merged PRs in 30d
Description
Summary
mape silently returns NaN when every target is (near-)zero. The near-zero filter then selects nothing, and np.mean of an empty slice yields NaN (plus a RuntimeWarning) instead of a clear error — while the scalar sibling ape asserts against exactly this case.
Location
- File:
causalml/metrics/regression.py - Functions:
mape(y, p)(primary);smape(y, p)has the analogous all-y == p == 0→0/0→ NaN path - Constant:
causalml/metrics/const.py,EPS = 1e-15
Relevant code path (static analysis of current main):
def mape(y, p):
filt = np.abs(y) > EPS
return np.mean(np.abs(1 - p[filt] / y[filt])) # mean of empty slice -> nan when filt is all False
def ape(y, p):
assert np.abs(y) > EPS # scalar version guards; vector version does not
return np.abs(1 - p / y)
Problem
All-zero (or all-sub-EPS) target vectors are realistic in uplift work — e.g. scoring a slice with zero conversions. mape then computes np.mean over an empty selection and returns nan, which propagates silently through model comparisons and leaderboards. The only signal is a numpy RuntimeWarning ("Mean of empty slice"), easily lost in training logs. smape shares the shape of the problem: with y == p == 0 everywhere, the ratio is 0/0 → NaN per element, and the mean is NaN.
Trigger / Reproduction
Based on static analysis (no execution performed):
import numpy as np
from causalml.metrics import mape
mape(np.zeros(100), np.zeros(100)) # filt all False -> np.mean([]) -> nan
Note: this is a static-analysis finding; I did not run the snippet, but the numpy empty-slice-means-nan semantics are documented behavior.
Expected Behavior
An explicit, actionable failure (e.g. ValueError when no target exceeds EPS) — or at minimum a causalml logger warning naming the function — consistent with ape's refusal to divide by ~zero. Never a silent NaN metric.
Actual Behavior
Returns NaN with only a numpy RuntimeWarning, indistinguishable downstream from a computed score.
Impact
- Silent leaderboard corruption: NaN sorts/aggregates unpredictably, and a "best model" pick can hinge on it.
- Inconsistency within the same module: scalar
apeguards, vectormape/smapedo not.
Suggested Direction
- Guard
mape(and thesmapeall-zero case) with an explicit check on the filtered selection, raisingValueErroror logging a warning and returning NaN deliberately — maintainer's choice, but make it explicit and documented in the docstrings.
Evidence
- Source via API:
regression.pymape/smape/apeas quoted;const.pyEPS. - Duplicate check: issue searches for
mape NaN empty(total_count: 0) andmetric nan zero(only unrelated closed feature requests #994, #887) — no apparent duplicate.
Classification
- FACT: all-~zero
ymakesmapetake the mean of an empty slice (verified in source via API). - INFERENCE: the result is NaN propagated as if it were a score.
- HYPOTHESIS: an explicit guard preserves all current behavior on valid inputs while making the degenerate case visible.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in causalml/metrics/regression.py by reading mape, smape, and ape, then check EPS in causalml/metrics/const.py and run the supplied zero-target reproduction. Add focused coverage for the degenerate cases and make the failure or warning explicit and documented, without changing valid-input behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100