AI-powered timeout failure categorization for prioritized investigation

## Summary

A significant portion of CI/CD failures are caused by timeouts. Today, these timeouts are reported as individual, unrelated failures — making it hard to prioritize which timeout patterns to investigate first. This feature adds **intelligent categorization of timeout failures into buckets** so that engineering teams can see aggregated statistics (e.g., "35% of timeout failures this week are Helix agent pool starvation on linux-arm64") and prioritize fixes by impact.

## Background

### The Problem

Timeouts are one of the most common failure modes in .NET CI/CD. They appear as:
- **Build step timeouts**: AzDO pipeline steps exceeding their configured timeout (e.g., "The job running on agent X has exceeded the maximum execution time")
- **Test timeouts**: Individual tests or Helix work items timing out (e.g., exit code -1 with no error message, or `Xunit.Sdk.TestTimeoutException`)
- **Infrastructure timeouts**: Agent connectivity, NuGet restore, Docker API timeouts, etc.
- **Canceled/Abandoned jobs**: Jobs that are canceled due to timeout cascades upstream

Examples from existing Known Issues in `dotnet/dnceng`:
- [#6452](https://github.com/dotnet/dnceng/issues/6452) — Generic "timeout"
- [#6414](https://github.com/dotnet/dnceng/issues/6414) — "Timeouts when running android-arm64 CoreCLR AllSubsets"
- [#6012](https://github.com/dotnet/dnceng/issues/6012) — "Linux-arm64 NativeAOT timeout"
- [#6004](https://github.com/dotnet/dnceng/issues/6004) — "runtime (Build Libraries Test Run) failed with timeout"
- [#4924](https://github.com/dotnet/dnceng/issues/4924) — "Tests failing due to Docker API timeout in Helix"
- [#4547](https://github.com/dotnet/dnceng/issues/4547) — "Infra timeout causing 'Unable to obtain kernel buffer'"

**The challenge**: These timeouts look different on the surface (different error messages, different test names, different platforms) but often share a common **root cause** (e.g., agent pool pressure, specific platform flakiness, resource exhaustion). Today there is no automated way to group them into actionable categories.

### What data is available

Build Insights already collects rich failure context during build analysis (`BuildAnalysisProvider.cs`):

| Data Source | Available Fields |
|-------------|-----------------|
| **Build step failures** (`StepResult`) | `StepName`, `Errors[].ErrorMessage`, `StepHierarchy`, `JobId`, `LinkLog` |
| **Test failures** (`TestCaseResult`) | `Name`, `ErrorMessage`, `StackTrace`, `DurationInMilliseconds`, `Comment` (Helix work item info) |
| **Helix work items** (`HelixWorkItem`) | `HelixJobId`, `HelixWorkItemName`, `ConsoleLogUrl`, `ExitCode`, `Status` |
| **Build metadata** (`Build`) | `DefinitionName`, `Repository.Name`, `TargetBranch`, `ProjectName` |
| **Timeline records** (`TimelineRecord`) | `Result` (Failed/Canceled/Abandoned), `RecordType`, `Name`, `StartTime` |
| **Kusto telemetry** | `KnownIssues` and `TestKnownIssues` tables with match history |

## Proposed Solution

### Approach: Embedding-Based Clustering with LLM-Assisted Labeling

Rather than a pure vector DB + search approach (which is better suited for retrieval of similar items), the recommended approach is a **two-phase system**:

#### Phase 1: Classification Pipeline (Batch/Offline)

1. **Timeout Detection**: Identify timeout failures from the signals already available in Build Insights:
- `TaskResult.Canceled` or `TaskResult.Abandoned` on timeline records
- Error messages containing timeout-related keywords (configurable list)
- Test results with high `DurationInMilliseconds` relative to historical baseline
- Helix work items with specific exit codes (e.g., -1, timeout-related codes)

2. **Feature Extraction**: For each timeout failure, extract a normalized feature vector from:
- Error message text (cleaned of build-specific identifiers like paths, IDs, timestamps)
- Step/test name and hierarchy
- Platform/configuration (extracted from step hierarchy or test name patterns)
- Repository and pipeline definition
- Duration and timing patterns

3. **Embedding & Clustering**: Generate embeddings for timeout failures and cluster them:
- **Option A — Text Embeddings + Clustering**: Use Azure OpenAI embeddings (`text-embedding-3-small`) on the normalized error context, then apply DBSCAN or HDBSCAN clustering. Store embeddings in a vector store (Azure AI Search, pgvector, or similar).
- **Option B — LLM-Based Classification** (simpler, recommended to start): Send batches of timeout error contexts to an LLM with a prompt like: *"Given these N timeout failures with their error messages, step names, and platforms, group them into categories with a descriptive label for each category."* This avoids the operational overhead of a vector DB and leverages the LLM's understanding of .NET CI/CD patterns.
- **Option C — Hybrid**: Use embeddings for similarity detection and pre-grouping, then LLM for labeling and merging similar clusters.

4. **Category Storage**: Store the resulting categories in Kusto (new `TimeoutCategories` table) with:
- `CategoryId` (stable identifier)
- `CategoryLabel` (human-readable, e.g., "Linux ARM64 agent pool timeout", "Docker API timeout in Helix")
- `CategoryDescription` (LLM-generated explanation of the root cause pattern)
- `MatchCount`, `DistinctBuilds`, `DistinctRepos`, `FirstSeen`, `LastSeen`
- `AffectedPlatforms`, `AffectedRepositories`

#### Phase 2: Real-Time Classification (Online)

Once categories are established, classify new timeout failures in real-time during build analysis:

1. When `BuildAnalysisProvider` processes a build and detects a timeout failure, compute its feature vector
2. Compare against known categories (nearest-neighbor search against category centroids or LLM classification against known categories)
3. Assign the failure to an existing category or flag it as "uncategorized" for the next batch run
4. Include the category information in the Build Analysis check output and Kusto telemetry

### Build Analysis Integration

#### Check Output Enhancement

Add a new section to the Build Analysis check for timeout failures:

> ### ⏱️ Timeout Failures
>
> | Category | Hits (this build) | Hits (last 7d) | Affected Repos | Trend |
> |----------|-------------------|-----------------|-----------------|-------|
> | Linux ARM64 agent pool timeout | 2 | 47 | runtime, aspnetcore | ↑ 23% |
> | Docker API timeout in Helix | 1 | 12 | runtime | ↓ 15% |
> | Uncategorized timeout | 1 | 3 | runtime | — |

#### Kusto Telemetry

New table `TimeoutFailures` for per-failure records:

```kusto
TimeoutFailures
| where Timestamp > ago(7d)
| summarize
HitCount = count(),
DistinctBuilds = dcount(BuildId),
DistinctRepos = dcount(Repository),
AvgDurationMs = avg(DurationMs)
by CategoryId, CategoryLabel
| order by HitCount desc
```

New table `TimeoutCategories` for the category definitions themselves (updated by the batch pipeline).

#### Dashboard / Reporting

Provide Kusto queries and/or a lightweight dashboard showing:
- **Top timeout categories** by frequency, trend, and blast radius
- **Category details**: which repos, platforms, pipelines, and time ranges are affected
- **Uncategorized timeouts**: failures that don't fit existing categories (candidates for new Known Issues or investigation)

## Implementation Considerations

### Where to run the classification

| Option | Pros | Cons |
|--------|------|------|
| **In BuildAnalysisProvider** (online) | Real-time, per-build categorization | Adds latency to build analysis; LLM calls during processing |
| **Separate batch job** (offline) | No latency impact on build analysis; can reprocess history | Categories lag behind real-time; needs separate scheduling |
| **Hybrid** | Best of both: batch for discovery, online for assignment | More complex architecture |

**Recommendation**: Start with **batch-only** (Phase 1) as a Kusto-based pipeline. Once categories stabilize, add lightweight online classification (Phase 2) using the established categories as a lookup table — no LLM call needed at build-analysis time for known categories.

### LLM vs. Vector DB

| Approach | When to use |
|----------|-------------|
| **LLM classification** (Option B) | Best for starting out. Simpler, no vector DB infra needed. Good when categories are relatively stable and the volume is manageable for batch LLM calls. |
| **Embedding + Vector DB** (Option A) | Better at scale. Useful if you want to build a "similarity search" experience (e.g., "find all failures similar to this one"). Requires vector store infra (Azure AI Search, pgvector). |
| **Hybrid** (Option C) | Best long-term. Use embeddings for fast pre-grouping, LLM for labeling and merging. |

### Error Normalization

Critical for good clustering — timeout error messages often contain build-specific noise:

```
# Before normalization:
"##[error]The job running on agent Hosted Agent 15 has exceeded the maximum execution time of 01:00:00"
"##[error]The job running on agent Hosted Agent 23 has exceeded the maximum execution time of 01:00:00"

# After normalization:
"The job running on agent [AGENT] has exceeded the maximum execution time of [DURATION]"
```

Normalization rules should strip:
- Agent names, machine names, IP addresses
- File paths, absolute paths
- Build IDs, run IDs, GUIDs
- Timestamps, durations (but preserve the fact that a duration exists)
- NuGet package versions

### Source Code Touchpoints

All code on the [`build-analysis` branch](https://github.com/dotnet/arcade-services/tree/build-analysis/src/BuildInsights):

| Component | File(s) | Change |
|-----------|---------|--------|
| Timeout detection | `BuildAnalysisProvider.cs` | Add timeout classification after failure analysis |
| Feature extraction | New: `TimeoutClassifier.cs` | Extract and normalize timeout features |
| Categorization | New: `TimeoutCategorizationService.cs` | LLM/embedding-based categorization logic |
| Kusto telemetry | `KnownIssuesProvider.cs` or new provider | Write `TimeoutFailures` / `TimeoutCategories` tables |
| Markdown output | `MarkdownGenerator.cs`, templates | Add timeout category section to check output |
| Batch job | New project or service | Periodic re-clustering of timeout failures |
| Configuration | `appsettings.Shared.json` | Azure OpenAI endpoint, model config, normalization rules |

## Phased Rollout

### Phase 0: Data Collection (Prerequisite)
- Add timeout detection heuristics to `BuildAnalysisProvider`
- Write raw timeout failure data to a new Kusto table (`TimeoutFailuresRaw`)
- No categorization yet — just capture the data for analysis

### Phase 1: Offline Categorization
- Build a batch job that reads from `TimeoutFailuresRaw`
- Run LLM-based categorization on batches of timeout failures
- Write categories to `TimeoutCategories` and assignments to `TimeoutFailures`
- Provide Kusto queries for manual analysis

### Phase 2: Build Analysis Integration
- Show timeout categories in the Build Analysis check output
- Classify new timeouts against known categories in real-time (lookup, no LLM call)
- Surface trends and affected repos

### Phase 3: Actionability
- Auto-suggest Known Issue creation for high-frequency timeout categories
- Link timeout categories to existing Known Issues
- Alerting on new/growing timeout categories

## Open Questions

1. **Azure OpenAI access**: Does the Build Insights service already have access to Azure OpenAI, or does this need to be provisioned?
2. **Cost**: What's the expected volume of timeout failures per day? This determines whether LLM-based classification is cost-effective at scale vs. embedding-only.
3. **Category stability**: Should categories be immutable once created, or should the system merge/split categories as patterns evolve?
4. **Historical backfill**: How far back should we categorize existing timeout failures? (Data is available in Kusto for the last 30 days via `KnownIssues` table.)
5. **Integration with Known Issues**: Should a timeout category automatically create a "tracking only" Known Issue (see #6139)?

## References

- **Source code (build-analysis branch):** https://github.com/dotnet/arcade-services/tree/build-analysis/src/BuildInsights
- [Build Analysis Introduction](https://github.com/dotnet/arcade/blob/main/Documentation/Projects/Build%20Analysis/Introduction.md)
- [Known Issues Documentation](https://github.com/dotnet/arcade/blob/main/Documentation/Projects/Build%20Analysis/KnownIssues.md)
- [Related: Tracking Only mode for Known Issues (#6139)](https://github.com/dotnet/arcade-services/issues/6139)
- [Known Issues Project Board](https://github.com/orgs/dotnet/projects/111)