dotnet / dotnet/msbuild

Ensure PerfStar ingests partial results after timeout cancellation

Open
#15,014 0 comments 0 reactions 0 assignees View on GitHub
Area: PerfStar
Dominant language
C#
Stars
5.5k
Forks
1.5k
Avg merge
1d 8h
Merged PRs (30d)
141

Description

**Problem**

Canceled PerfStar runs do not send completed performance data to Kusto.

Runs 15296191 and 15296217 reached the 240-minute job timeout. Azure Pipelines then canceled both GOLD jobs.

The `Ingest GOLD metrics to Kusto` step uses `succeededOrFailed()`. Azure Pipelines skips this condition after job cancellation.

The later artifact steps use `always()`. These steps published partial results to GitHub, but Kusto received no performance rows.

The comparison website reads `PerfStarDataRaw` from Kusto. It does not use the published GitHub dataset registry.

**Desired outcome**

All Live PerfStar runs must send completed performance data to Kusto. This requirement includes runs that fail, time out, or are canceled.

The comparison website must show a branch when its run produced valid performance data.

**Possible solutions**

1. Change the Kusto ingestion condition from `succeededOrFailed()` to `always()`.
2. Add `cancelTimeoutInMinutes` to give cleanup steps sufficient time.
3. Give the test step a shorter timeout than the job. The remaining job time can then run ingestion and artifact steps.
4. Move ingestion to a separate job. That job can download the raw-results artifact after the collection job stops.

The first three changes are the smallest solution. A separate ingestion job gives better isolation from collection-job cancellation.

### Completion criteria

- A timed-out GOLD job sends all completed scenario measurements to `PerfStarDataRaw`.
- A canceled GOLD job sends all completed scenario measurements to `PerfStarDataRaw`.
- A failed GOLD job continues to send all completed scenario measurements.
- Missing processor output causes a safe no-op result.
- A missing results directory causes a safe no-op result.
- An empty result set causes a safe no-op result.
- The pipeline reserves sufficient time for ingestion after test execution stops.
- An automated test verifies the timeout or cancellation path.
- The comparison catalog includes a branch after Kusto receives valid rows.

### Dependencies and handoff

Observed runs:

- Configuration logging: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=15296217
- Composite globs: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=15296191

Relevant files in the internal `dotnet-perfstar` repository:

- `pipeline/perf-collection-job.yml`
- `src/RunResultsProcessor/Commands/EmitGoldKustoRowsCommand.cs`
- `src/RunResultsProcessor/Commands/PushToKustoCommand.cs`
- `dashboard/PerfStar.Api/Comparison/ComparisonCatalogQuery.cs`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with pipeline/perf-collection-job.yml to trace cancellation and ingestion conditions, then read EmitGoldKustoRowsCommand.cs, PushToKustoCommand.cs, and ComparisonCatalogQuery.cs. Check how partial, missing, or empty results are handled and identify the existing pipeline test entry point. Done means failed, timed-out, and canceled GOLD runs safely ingest completed rows and the comparison catalog includes valid branches.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp, github
Domain
backend-api-design, ci-cd, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.