'Tracking Only' mode for Known Issues - track statistics without excusing failures

## Summary

Introduce a **"Tracking Only"** mode for Known Issues so that Build Insights (formerly Build Analysis) can match and collect statistics for a known failure **without** turning the Build Analysis check green. This decouples failure tracking from failure exemption.

## Background

### Current Behavior

[Known Issues](https://github.com/dotnet/arcade/blob/main/Documentation/Projects/Build%20Analysis/KnownIssues.md) are GitHub issues labeled `Known Build Error` that contain a JSON blob describing an error pattern:

```json
{
"ErrorMessage": "The Operation will be canceled. The next steps may not contain expected logs",
"BuildRetry": false,
"ErrorPattern": "",
"ExcludeConsoleLog": false
}
```

When Build Analysis processes a PR build and **all** failures match open Known Issues, the Build Analysis check turns **green (success)**, effectively excusing those failures and allowing the PR to merge. Telemetry is also sent to Kusto (`KnownIssues` / `TestKnownIssues` tables in `engineeringdata`).

**Example known issue:** [dotnet/dnceng#6408](https://github.com/dotnet/dnceng/issues/6408)

### The Problem

Currently, creating a Known Issue is **all-or-nothing**: it always turns the check green when matched. There is no way to:

1. **Collect statistics** about a known failure's frequency and blast radius without also excusing it
2. **Keep the check red** to signal that the failure still needs attention, even though it's a known/tracked issue
3. **Gradually manage** a known issue — e.g., start by tracking it, then later decide to excuse it

This forces teams into an uncomfortable choice:
- **Create a Known Issue** → check turns green, PR can merge, but the failure is silently excused and teams stop feeling urgency to fix it
- **Don't create a Known Issue** → no centralized tracking, no telemetry, developers waste time investigating failures that are already understood

### Real-World Scenarios

- **New intermittent failure discovered**: A team wants to track how often it hits across repos and builds before deciding whether it should be excused. With "tracking only," they get Kusto telemetry and Build Analysis annotations without masking the failure.
- **High-severity known issue**: A failure is known and being actively worked on, but the team does **not** want it excused — PRs hitting this issue should not merge until it's fixed. They still want centralized tracking.
- **Issue under investigation**: Engineering services needs data on frequency/scope of a failure to prioritize it, but doesn't want to give repos a free pass while investigating.

## Proposed Solution

### New JSON Property: `TrackingOnly`

Add a new boolean property `TrackingOnly` (default: `false`) to the Known Issue JSON schema:

```json
{
"ErrorMessage": "The Operation will be canceled. The next steps may not contain expected logs",
"BuildRetry": false,
"ErrorPattern": "",
"ExcludeConsoleLog": false,
"TrackingOnly": true
}
```

### Behavior when `TrackingOnly: true`

| Aspect | Current (TrackingOnly=false) | Proposed (TrackingOnly=true) |
|--------|------|------|
| Error matching | ✅ Matches failures against Known Issue | ✅ Same — still matches |
| Build Analysis display | ✅ Shows "Known Issue" annotation | ✅ Same — still shows annotation (with distinct visual) |
| Kusto telemetry | ✅ Records match in `KnownIssues`/`TestKnownIssues` | ✅ Same — still records (with `TrackingOnly` flag) |
| Check conclusion | 🟢 Turns green if all failures matched | 🔴 **Stays red** — failure is NOT excused |
| Build retry | Respects `BuildRetry` flag | ❌ **No retry** — `BuildRetry` is ignored when `TrackingOnly=true` |
| PR merge | ✅ Allowed (check is green) | ❌ **Blocked** (check stays red) |

### Build Analysis Display

When a failure matches a "tracking only" Known Issue, Build Analysis should clearly indicate the distinction in its output, e.g.:

> ⚠️ **Known Issue (Tracking Only):** [dotnet/dnceng#6408](https://github.com/dotnet/dnceng/issues/6408) — *[android-arm64] The Operation will be canceled...*
> This failure matches a known issue that is being tracked for statistics. The Build Analysis check remains failed.

This contrasts with the current display for regular known issues:

> ✅ **Known Issue:** [dotnet/dnceng#6408](https://github.com/dotnet/dnceng/issues/6408) — *[android-arm64] The Operation will be canceled...*

### Telemetry

The Kusto telemetry tables (`KnownIssues` / `TestKnownIssues`) should include a new `TrackingOnly` boolean column, enabling queries like:

```kusto
KnownIssues
| where TrackingOnly == true
| summarize HitCount=count(), DistinctBuilds=dcount(BuildId) by IssueId, IssueRepository
| order by HitCount desc
```

### Known Issue Validation

The existing Known Issue validation bot (`KnownIssueValidationProvider.cs`) should also validate `TrackingOnly` issues and mention the mode in its validation comment, e.g.:

> ✅ Known issue matched with the provided build. **Mode: Tracking Only** (failures will be tracked but not excused).

## Implementation Plan

All code is on the [`build-analysis` branch](https://github.com/dotnet/arcade-services/tree/build-analysis/src/BuildInsights) of `dotnet/arcade-services`.

### 1. JSON Schema — `KnownIssueJson.cs`

**File:** [`src/BuildInsights/BuildInsights.KnownIssues/Models/KnownIssueJson.cs`](https://github.com/dotnet/arcade-services/blob/build-analysis/src/BuildInsights/BuildInsights.KnownIssues/Models/KnownIssueJson.cs)

Add `TrackingOnly` property:

```csharp
public class KnownIssueJson
{
[JsonConverter(typeof(ErrorOrArrayOfErrorsConverter))]
public List ErrorMessage { get; set; } = [];

[JsonConverter(typeof(ErrorOrArrayOfErrorsConverter))]
public List ErrorPattern { get; set; } = [];

public bool BuildRetry { get; set; }
public bool ExcludeConsoleLog { get; set; }
public bool TrackingOnly { get; set; } // ← NEW
}
```

### 2. Options Model — `KnownIssueOptions.cs`

**File:** [`src/BuildInsights/BuildInsights.KnownIssues/Models/KnownIssueOptions.cs`](https://github.com/dotnet/arcade-services/blob/build-analysis/src/BuildInsights/BuildInsights.KnownIssues/Models/KnownIssueOptions.cs)

Propagate `TrackingOnly` through options:

```csharp
public class KnownIssueOptions
{
public bool ExcludeConsoleLog { get; }
public bool RetryBuild { get; }
public bool RegexMatching { get; }
public bool TrackingOnly { get; } // ← NEW

public KnownIssueOptions(
bool excludeConsoleLog = default,
bool retryBuild = default,
bool regexMatching = default,
bool trackingOnly = default) // ← NEW
{
ExcludeConsoleLog = excludeConsoleLog;
RetryBuild = retryBuild;
RegexMatching = regexMatching;
TrackingOnly = trackingOnly;
}
}
```

### 3. Issue Parser — `KnownIssueHelper.cs`

**File:** [`src/BuildInsights/BuildInsights.KnownIssues/KnownIssueHelper.cs`](https://github.com/dotnet/arcade-services/blob/build-analysis/src/BuildInsights/BuildInsights.KnownIssues/KnownIssueHelper.cs)

In `ParseGithubIssue()`, pass `TrackingOnly` through to `KnownIssueOptions`:

```csharp
// Current:
new KnownIssueOptions(knownIssueJson.ExcludeConsoleLog, knownIssueJson.BuildRetry, regexMatching: true)

// Updated:
new KnownIssueOptions(knownIssueJson.ExcludeConsoleLog, knownIssueJson.BuildRetry, regexMatching: true, trackingOnly: knownIssueJson.TrackingOnly)
```

Also update `GetKnownIssueSectionTemplate()` and `GetKnownIssueJsonFilledIn()` to include `TrackingOnly` in the template JSON.

### 4. Check Conclusion Logic — `CheckResultProvider.cs` ⚠️ **CRITICAL**

**File:** [`src/BuildInsights/BuildInsights.BuildAnalysis/CheckResultProvider.cs`](https://github.com/dotnet/arcade-services/blob/build-analysis/src/BuildInsights/BuildInsights.BuildAnalysis/CheckResultProvider.cs)

This is the core logic change. The `GetBuildStatusWithKnownIssues()` method currently determines if failures should be excused:

```csharp
// CURRENT LOGIC:
bool hasUniqueBuildFailures = stepResults.Any(t => t.KnownIssues.Count == 0);
bool hasUniqueTestFailures = testResults.Any(t => t.TestCaseResult.Outcome == TestOutcomeValue.Failed &&
t.KnownIssues.Count == 0 && !t.IsKnownIssueFailure);
```

**Change:** A failure matched _only_ by tracking-only known issues should still count as "unique" (not excused). Only non-tracking-only known issues can excuse a failure:

```csharp
// PROPOSED LOGIC:
// A step is "unique" (not excused) if it has no known issues, OR all its known issues are tracking-only
bool hasUniqueBuildFailures = stepResults.Any(t =>
t.KnownIssues.Count == 0 ||
t.KnownIssues.All(ki => ki.Options.TrackingOnly));

bool hasUniqueTestFailures = testResults.Any(t =>
t.TestCaseResult.Outcome == TestOutcomeValue.Failed &&
!t.IsKnownIssueFailure &&
(t.KnownIssues.Count == 0 ||
t.KnownIssues.All(ki => ki.Options.TrackingOnly)));
```

**Precedence rule:** If a failure matches BOTH a regular and a tracking-only known issue, the regular one takes precedence and the failure is excused. This is handled naturally — as long as at least one non-tracking-only issue matches, the `All(ki => ki.Options.TrackingOnly)` check will be `false`.

### 5. Build Retry — `BuildRetryProvider.cs`

**File:** [`src/BuildInsights/BuildInsights.BuildAnalysis/BuildRetryProvider.cs`](https://github.com/dotnet/arcade-services/blob/build-analysis/src/BuildInsights/BuildInsights.BuildAnalysis/BuildRetryProvider.cs)

When evaluating whether to retry a build, skip known issues that have `TrackingOnly = true`. A tracking-only issue should never trigger an automatic retry.

### 6. Kusto Telemetry — `KnownIssuesProvider.cs`

**File:** [`src/BuildInsights/BuildInsights.KnownIssues/KnownIssuesProvider.cs`](https://github.com/dotnet/arcade-services/blob/build-analysis/src/BuildInsights/BuildInsights.KnownIssues/KnownIssuesProvider.cs)

Add `TrackingOnly` column to both Kusto mapping methods:

```csharp
// In MapKnownIssueMatch(KnownIssueMatch match):
new KustoValue("TrackingOnly", match.TrackingOnly, KustoDataType.Boolean),

// In MapKnownIssueMatch(TestKnownIssueMatch match):
new KustoValue("TrackingOnly", match.TrackingOnly, KustoDataType.Boolean),
```

### 7. Match Models — `KnownIssueMatch.cs` / `TestKnownIssueMatch.cs`

**Files:**
- [`src/BuildInsights/BuildInsights.KnownIssues/Models/KnownIssueMatch.cs`](https://github.com/dotnet/arcade-services/blob/build-analysis/src/BuildInsights/BuildInsights.KnownIssues/Models/KnownIssueMatch.cs)
- [`src/BuildInsights/BuildInsights.KnownIssues/Models/TestKnownIssueMatch.cs`](https://github.com/dotnet/arcade-services/blob/build-analysis/src/BuildInsights/BuildInsights.KnownIssues/Models/TestKnownIssueMatch.cs)

Add `public bool TrackingOnly { get; set; }` to both models.

### 8. Match Helper — `KnownIssuesMatchHelper.cs`

**File:** [`src/BuildInsights/BuildInsights.BuildAnalysis/KnownIssuesMatchHelper.cs`](https://github.com/dotnet/arcade-services/blob/build-analysis/src/BuildInsights/BuildInsights.BuildAnalysis/KnownIssuesMatchHelper.cs)

Populate `TrackingOnly` when creating match objects:

```csharp
// In GetKnownIssueMatchesInBuild:
TrackingOnly = issue.Options.TrackingOnly,

// In GetKnownIssueMatchesInTests:
TrackingOnly = issue.Options.TrackingOnly,
```

### 9. Markdown / Handlebars Templates

**Directory:** [`src/BuildInsights/BuildInsights.BuildAnalysis/Templates/`](https://github.com/dotnet/arcade-services/tree/build-analysis/src/BuildInsights/BuildInsights.BuildAnalysis/Templates)

Update the Handlebars templates and `MarkdownGenerator.cs` to display tracking-only known issues differently (e.g., ⚠️ icon, "(Tracking Only)" suffix, explanatory text).

### 10. Known Issue Validation — `KnownIssueValidationProvider.cs`

**File:** [`src/BuildInsights/BuildInsights.BuildAnalysis/KnownIssueValidationProvider.cs`](https://github.com/dotnet/arcade-services/blob/build-analysis/src/BuildInsights/BuildInsights.BuildAnalysis/KnownIssueValidationProvider.cs)

Include the tracking-only mode in the validation comment posted to the GitHub issue.

## Edge Cases

| Scenario | Expected Behavior |
|----------|-------------------|
| All failures match, but all are tracking-only | Check stays **red** — tracking-only matches don't excuse failures |
| Mix of regular and tracking-only matches | Only regular matches excuse their failures. If any unexcused failures remain, check stays red |
| Same failure matches both a regular AND tracking-only issue | Regular known issue takes precedence — failure is **excused** |
| `BuildRetry: true` + `TrackingOnly: true` | `BuildRetry` is **ignored** — no retry is triggered |
| `TrackingOnly: true` on a repo with `ShouldMergeOnFailureWithKnownIssues: false` | No behavioral change — the check was already not turning green for known issues in that repo |

## Migration & Backward Compatibility

- **No migration needed** — existing issues without `TrackingOnly` default to `false`, preserving current behavior
- **No Kusto schema migration blocking** — the new `TrackingOnly` column can be added as a nullable boolean; existing rows will have `null` (treated as `false`)
- Teams can opt-in by adding `"TrackingOnly": true` to existing or new known issues

## Open Questions

1. **Naming**: Is `TrackingOnly` the best name? Alternatives: `StatisticsOnly`, `DoNotExcuse`, `Monitor`, `TrackWithoutExcusing`
2. **UI in Build Analysis**: Should tracking-only matches be displayed in a separate section from regular known issues, or inline with a different icon?
3. **Interaction with `BuildRetry`**: Should `BuildRetry: true` + `TrackingOnly: true` be a validation error, or should `TrackingOnly` silently override `BuildRetry`?
4. **Known Issues Board**: Should the [Known Issues project board](https://github.com/orgs/dotnet/projects/111) have a separate view/column for tracking-only issues?
5. **Converting between modes**: Is editing the JSON blob in the GitHub issue sufficient, or should there be a streamlined toggle mechanism?

## References

- **Source code (build-analysis branch):** https://github.com/dotnet/arcade-services/tree/build-analysis/src/BuildInsights
- [Build Analysis Introduction](https://github.com/dotnet/arcade/blob/main/Documentation/Projects/Build%20Analysis/Introduction.md)
- [Known Issues Documentation](https://github.com/dotnet/arcade/blob/main/Documentation/Projects/Build%20Analysis/KnownIssues.md)
- [Known Issue JSON Step-by-Step](https://github.com/dotnet/arcade/blob/main/Documentation/Projects/Build%20Analysis/KnownIssueJsonStepByStep.md)
- [Example Known Issue: dotnet/dnceng#6408](https://github.com/dotnet/dnceng/issues/6408)
- [Known Issues Project Board](https://github.com/orgs/dotnet/projects/111)
- [DNCEng Services Wiki: Build Analysis](https://dnceng.visualstudio.com/internal/_wiki/wikis/DNCEng%20Services%20Wiki/547/Build-Analysis)