Monitoring metrics crash on NaN/Inf feature values, silently dropping the whole feature view
- Dominant language
- Python
- Stars
- 7.3k
- Forks
- 1.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 15
Description
## Expected Behavior
Computing monitoring metrics for a numeric feature should succeed even when some values are NaN or +-Infinity, which occur naturally in feature engineering (e.g. a ratio whose denominator is zero).
## Current Behavior
`MetricsCalculator.compute_numeric()` passes values straight into `np.histogram()`, which raises on non-finite input:
ValueError: autodetected range of [0.05, inf] is not finite
`compute_all()` has no per-column error handling, so this exception discards metrics for the entire feature view - including columns that are perfectly well-formed.
The failure is silent to the caller: POST /monitoring/compute (and the auto_compute endpoint the Feast UI's "Compute Metrics" button calls) still returns HTTP 200 with "status": "completed" and "computed_features": 0. Nothing surfaces to the UI - the feature view is simply absent from the Monitoring page, with no error shown anywhere.
The same unguarded np.histogram call is duplicated in the Dask offline store (_dask_compute_numeric_metrics), so it fails there too.
## Steps to reproduce
import pyarrow as pa
from feast.monitoring.metrics_calculator import MetricsCalculator
calc = MetricsCalculator()
arr = pa.array([0.05, 0.06, float("inf"), 0.08], type=pa.float64())
calc.compute_numeric(arr)
# ValueError: autodetected range of [0.05, inf] is not finite
Or end-to-end: define a FeatureView with a feature computed as clicks / impressions where one row has impressions = 0, enable data_quality_monitoring in feature_store.yaml, and run feast apply. The apply succeeds but logs:
ERROR:feast.monitoring.monitoring_service:Failed to compute baseline for feature view 'campaign_stats'
ValueError: autodetected range of [0.05, inf] is not finite
and the feature view never appears on the Monitoring page, even though feast apply itself reports no error.
### Specifications
- Version: master
- Platform: Linux (also affects the Dask offline store code path)
- Subsystem: monitoring / metrics_calculator
## Possible Solution
Filter non-finite values out after dropping nulls, before computing any statistic, and apply the existing _safe_float/opt_float helpers (which already exist for this exact purpose but were only applied to mean and stddev) to min_val, max_val, and the quantiles as well. Fix in both feast/monitoring/metrics_calculator.py and feast/infra/offline_stores/dask.py. Submitted as a PR alongside this issue.
FIX :- #6782
Contributor guide
Research direction
Read feast/monitoring/metrics_calculator.py, especially MetricsCalculator.compute_numeric() and compute_all(), then inspect feast/infra/offline_stores/dask.py for _dask_compute_numeric_metrics. Verify behavior with the reproduction values containing NaN or infinity; done means both paths compute metrics for finite values without dropping the whole feature view and apply the existing safe numeric helpers to all reported statistics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data-engineering, observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100