NaN Propagation Causes SIGSEGV in Autodetect Process

# NaN Propagation Causes SIGSEGV in Autodetect Process

## Summary

The autodetect process crashes with SIGSEGV (signal 11) due to NaN values propagating through probability calculations, corrupting model weights, and leading to unsafe access of an empty accumulator. The root cause is extreme metric values (particularly `2^64` overflow values) causing mathematical overflow in variance/statistical calculations.

## Error Chain (Chronological Order)

1. **14:10:56** - NaN Probability Bounds Error (`COneOfNPrior.cc:960`): `Bad probability bounds = [-nan, -nan]`
2. **14:11:03** - Update Failed (`COneOfNPrior.cc:464`): `Update failed` with NaN log weights: `Update failed ( -2.596754686508612e+00 -nan -2.079860261157839e+01 )`
3. **14:11:46** - Bad Category (`CXMeansOnline1d.cc:468`): NaN scale parameter in normal distribution: `Bad category = (0.499001, 1.84467e+19, 0): Error in function boost::math::normal_distribution::normal_distribution: Scale parameter is -nan, but must be > 0 !`
4. **14:11:46** - Invalid Split (`CXMeansOnline1d.cc:530`): `Expected 2-split: []` (empty candidate)
5. **14:17:26** - Process Crash (Signal 11 SIGSEGV): Child process terminated

## Root Cause Analysis

### Issue 1: NaN Propagation in Probability Calculation

**Location**: `lib/maths/common/COneOfNPrior.cc:893-977`

**Problem**:
- Function `probabilityOfLessLikelySamples()` accumulates probability bounds from multiple models
- If `model.probabilityOfLessLikelySamples()` returns `true` but with NaN values in `modelLowerBound` or `modelUpperBound`, these NaN values propagate through accumulation (lines 953-954)
- Error is logged at line 960 but NaN values are only corrected AFTER the error log (lines 964-969)
- The NaN values propagate to other parts of the system

**Code Flow**:
```cpp
Line 942: model.probabilityOfLessLikelySamples() returns true with NaN values
Line 953-954: NaN values accumulated into lowerBound/upperBound
Line 960: ERROR logged: "Bad probability bounds = [-nan, -nan]"
Line 964-969: NaN corrected to default values (too late)
```

### Issue 2: NaN Propagation to Model Weights

**Location**: `lib/maths/common/COneOfNPrior.cc:463-468`

**Problem**:
- After probability calculation returns with NaN values, the model weights become corrupted
- `badWeights()` checks if log weights are finite (line 1147)
- When NaN values are present, `badWeights()` returns true
- Error logged: "Update failed" with debug output showing NaN weights
- Model is reset to non-informative state (line 467), but damage already done

### Issue 3: NaN Propagation to Clustering Algorithm

**Location**: `lib/maths/common/CXMeansOnline1d.cc:430-468`

**Problem**:
- NaN values propagate to category variance calculations
- `CBasicStatistics::maximumLikelihoodVariance(category)` returns NaN
- `sigma = std::sqrt(NaN)` → NaN
- `boost::math::normal_distribution(m, NaN)` throws exception

**Code Flow**:
```cpp
Line 430: sigma = std::sqrt(CBasicStatistics::maximumLikelihoodVariance(category));
Line 441: boost::math::normal normal(m, sigma); // sigma is NaN → exception
Line 468: ERROR logged: "Bad category = (0.499001, 1.84467e+19, 0): Error..."
```

### Issue 4: Corrupted State Causes Invalid Split

**Location**: `lib/maths/common/CXMeansOnline1d.cc:525-530`

**Problem**:
- After exception handling, algorithm state is corrupted
- Split candidate calculation fails, returns empty vector
- **Error**: "Expected 2-split: []"

### Issue 5: Unsafe Access to Empty Accumulator → SIGSEGV

**Location**: `lib/maths/common/COneOfNPrior.cc:972` (now 1001 after fixes)

**Problem**:
- Code accesses `tail_[0].second` without checking if `tail_` accumulator is empty
- `TDoubleTailPrMaxAccumulator` (which is `COrderStatisticsStack`) can be empty if:
- Loop never executes (e.g., `logWeights` is empty after corruption)
- Loop exits early via break condition before any `tail_.add()` calls
- NaN values cause early loop termination
- When `count() == 0`, accessing `tail_[0]` uses `m_Statistics[m_UnusedCount + 0]` where `m_UnusedCount == N`, causing out-of-bounds access → **SIGSEGV**

**Vulnerable Code**:
```cpp
Line 927: TDoubleTailPrMaxAccumulator tail_;
Line 928-956: Loop that may not add anything to tail_ (due to NaN/corruption)
Line 972: tail = tail_[0].second; // UNSAFE - no check for empty!
```

## Root Cause: Extreme Values in Production Data

Analysis of production data shows:
- **Max values**: `18446744073709551616` = `2^64` = `UINT64_MAX` (integer overflow indicators)
- **Huge standard deviations**: e.g., `3217383803720871424.000000` vs mean `578927774067923328.000000`
- **Negative values**: Some partitions have negative min values (`-13034`, `-433`, etc.)

These extreme values cause:
1. Mathematical overflow in variance calculations → NaN
2. NaN propagation through probability calculations
3. Corrupted model weights
4. Empty `normalizedLogWeights()` when all weights are NaN
5. Empty accumulator → SIGSEGV

## Proposed Fixes

1. **Early NaN Detection**: Check for NaN values immediately after `model.probabilityOfLessLikelySamples()` returns, before accumulation (lines 942-948)
2. **Empty Accumulator Guard**: Add check `if (tail_.count() > 0)` before accessing `tail_[0]` (line 972)
3. **Better Error Handling**: Return false if NaN detected or accumulator is empty, rather than continuing with corrected values
4. **NaN Checks in Clustering**: Add NaN validation before creating distribution objects in `CXMeansOnline1d.cc:430-441`
5. **Validate Split Candidates**: Add check for empty candidate vector before using it (line 529)

## Files Affected

- `lib/maths/common/COneOfNPrior.cc` (lines 893-977, especially 942-972)
- `lib/maths/common/CXMeansOnline1d.cc` (lines 430-441, 529-530)

## Production Workaround

Add datafeed query filters to exclude invalid values:

```json
{
"datafeed_config": {
"query": {
"bool": {
"must": [
{
"range": {
"field_value": {
"gte": -1e15,
"lte": 1e15
}
}
},
{
"script": {
"script": {
"source": "doc['field_value'].value != null && Double.isFinite(doc['field_value'].value)",
"lang": "painless"
}
}
}
]
}
}
}
}
```

This filters out:
- `2^64` overflow values (`1.84e19` > `1e15`)
- NaN/Infinity values
- Extreme outliers causing mathematical overflow

## Impact

- **Severity**: Critical (SIGSEGV crash)
- **Frequency**: Occurs when processing data with extreme outliers (2^64 values)
- **Affected**: Jobs processing metric data with integer overflow values
- **Workaround Available**: Yes (query filters)

## Additional Context

Error log excerpt:
```
ERROR [autodetect/230997] [[COneOfNPrior.cc]@960] Bad probability bounds = [-nan, -nan], [(-nan, 2), (-nan, 1), (-nan, 0)] -
ERROR [autodetect/230997] [[COneOfNPrior.cc]@464] Update failed ( -2.596754686508612e+00 -nan -2.079860261157839e+01 ) -
ERROR [autodetect/230997] [[CXMeansOnline1d.cc]@468] Bad category = (0.499001, 1.84467e+19, 0): Error in function boost::math::normal_distribution::normal_distribution: Scale parameter is -nan, but must be > 0 ! -
ERROR [controller/136] [[CDetachedProcessSpawner.cc]@201] Child process with PID 230997 was terminated by signal 11 Please check system logs for more details. -
```