dotnet / dotnet/aspnetcore

Skipped all of Templates.Mvc.Tests in Helix on Debian12 and Mariner

Open
#56,450 0 comments 0 reactions 0 assignees View on GitHub
area-mvc test-failure
Dominant language
C#
Stars
38.4k
Forks
10.9k
Avg merge
2d 10h
Merged PRs (30d)
281

Description

## Failing Test(s)

- All of Templates.Mvc.Tests in Helix on Debian12 and Mariner

See https://github.com/dotnet/aspnetcore/pull/56291 for more info and discussion.

## Error Message

```text
The Helix Work Item failed. Often this is due to a test crash. Please see the 'Artifacts' tab above for additional logs.
```

## Stacktrace

None

## Logs

Unfortunately we often don't get any, and instead this is what the CI system produces:

```text
If you’re reading this, that means the Helix work item you’re trying to find the logs for has dead-lettered.

What this means:

- All attempts to retry execution of this work item were unable to complete. This can be both for infrastructure reasons (problems within Azure) or issues with the work item (for instance, causing a machine to reboot unexpectedly or killing the Helix client on the machine will force a retry).
- No further work will be done for this specific work item, and its exit code is set to an artificial -1 (since it did not complete, there is no real exit code).

Common causes:

- Disabled queue (end-of-life Helix queues are automatically forwarded to deadletter and will fail instantly)
- Unhealthy Helix Client machine(s)
- Queue has been backed up heavily by a large amount of work and was manually purged by the engineering team
- Azure issues (e.g. Service Bus is overloaded)
- Malformed payloads; if Helix cannot download and unzip all payloads successfully, work will retry until dead-lettered.

For follow up:

- Check if your Helix Queue is still enabled, either via the metadata you see by browsing to https://helix.dot.net/api/info/queues?api-version=2019-06-17 or recent emails from the .NET Engineering Infrastructure team.
- Check that all work item payloads are accessible using a browser.
- If you are sending to a non-disabled queue and find this error repeatedly occurring, please contact the dnceng team.
- If a single, specific work item dead letters and others do not, consider local debugging; it may be causing spontaneous reboot (or trigging one intentionally).

```

## Build

https://github.com/dotnet/aspnetcore/runs/26663284864

Contributor guide

Open the contributing guide

Research direction

Review PR #56291 and the artifacts for build run 26663284864 first, focusing on why all Templates.Mvc.Tests work items were skipped on Debian 12 and Mariner. Confirm whether the Helix queues and payloads are available, then reproduce or inspect the failure. Done means the test suite runs successfully on both environments without dead-lettering.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, debian
Domain
ci-cd, infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.