feast-dev / feast-dev/feast

Monitoring metrics crash on NaN/Inf feature values, silently dropping the whole feature view

Open
#6,783 0 comments 0 reactions 0 assignees View on GitHub
kind/bug priority/p2
Dominant language
Python
Stars
7.3k
Forks
1.4k
Avg merge
1d 21h
Merged PRs (30d)
15

Description

## Expected Behavior

Computing monitoring metrics for a numeric feature should succeed even when some values are NaN or +-Infinity, which occur naturally in feature engineering (e.g. a ratio whose denominator is zero).

## Current Behavior

`MetricsCalculator.compute_numeric()` passes values straight into `np.histogram()`, which raises on non-finite input:

ValueError: autodetected range of [0.05, inf] is not finite

`compute_all()` has no per-column error handling, so this exception discards metrics for the entire feature view - including columns that are perfectly well-formed.

The failure is silent to the caller: POST /monitoring/compute (and the auto_compute endpoint the Feast UI's "Compute Metrics" button calls) still returns HTTP 200 with "status": "completed" and "computed_features": 0. Nothing surfaces to the UI - the feature view is simply absent from the Monitoring page, with no error shown anywhere.

The same unguarded np.histogram call is duplicated in the Dask offline store (_dask_compute_numeric_metrics), so it fails there too.

## Steps to reproduce

import pyarrow as pa
from feast.monitoring.metrics_calculator import MetricsCalculator

calc = MetricsCalculator()
arr = pa.array([0.05, 0.06, float("inf"), 0.08], type=pa.float64())
calc.compute_numeric(arr)
# ValueError: autodetected range of [0.05, inf] is not finite

Or end-to-end: define a FeatureView with a feature computed as clicks / impressions where one row has impressions = 0, enable data_quality_monitoring in feature_store.yaml, and run feast apply. The apply succeeds but logs:

ERROR:feast.monitoring.monitoring_service:Failed to compute baseline for feature view 'campaign_stats'
ValueError: autodetected range of [0.05, inf] is not finite

and the feature view never appears on the Monitoring page, even though feast apply itself reports no error.

### Specifications

- Version: master
- Platform: Linux (also affects the Dask offline store code path)
- Subsystem: monitoring / metrics_calculator

## Possible Solution

Filter non-finite values out after dropping nulls, before computing any statistic, and apply the existing _safe_float/opt_float helpers (which already exist for this exact purpose but were only applied to mean and stddev) to min_val, max_val, and the quantiles as well. Fix in both feast/monitoring/metrics_calculator.py and feast/infra/offline_stores/dask.py. Submitted as a PR alongside this issue.

FIX :- #6782

Contributor guide

Open the contributing guide

Research direction

Read feast/monitoring/metrics_calculator.py, especially MetricsCalculator.compute_numeric() and compute_all(), then inspect feast/infra/offline_stores/dask.py for _dask_compute_numeric_metrics. Verify behavior with the reproduction values containing NaN or infinity; done means both paths compute metrics for finite values without dropping the whole feature view and apply the existing safe numeric helpers to all reported statistics.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data-engineering, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.