llvm / llvm/offload-test-suite

windows-amd-dxc-d3d12 fails ~50% of the time in post-commit run with 'Failed to create PSO' across many tests

Open
#1,210 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

driver-bug needs-triage
Dominant language
C++
Stars
18
Forks
39
Avg merge
2d 18h
Merged PRs (30d)
40

Description

`windows-amd-dxc-d3d12` has been failing roughly half of post-commit runs. 16 of the last 30 scheduled runs failed. When we manually re-ran the workflow 8 times against a single SHA to characterize the flakiness, each run produced a different set of failing tests — so this is a race, not a regression.

In the post-commit pipeline, all failures look the same:
```
gpu-exec: error: Failed to create PSO
```
Exit 1, no crash signature in the captured logs. The pre-commit pipeline has separately been seeing actual crashes originating from the same `CreatePSO` call. We don't yet have an explanation for why the two pipelines surface differently.

Runner: RX 9070 / driver 32.0.31007.1017 (from the dxdiag artifact).

Tests run in parallel via lit, each in its own `offloader.exe` process. The root cause is still unclear — it's possible there's a race condition in the AMD UMD around `CreateComputePipelineState`.

Next steps I'd like to take:
1. Enable WER LocalDumps on the runner so the next pre-commit crash captures a user-mode dump. This needs a remote/admin session — it can't be done from a workflow.
2. Get an RX 9070 + matching driver into a dev box for local investigation — can't reproduce on RX 6800 / 32.0.21043.10005, and having admin + free re-run cycles would unblock most things.
3. Longer-term: kernel-mode debugging if the user-mode dump isn't enough.

Per-test frequency and per-run timeline in a comment below.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the post-commit and pre-commit runs using the per-test frequency and timeline from the issue comment, then inspect the gpu-exec CreatePSO failures across the lit-launched offloader.exe processes. Enable WER LocalDumps on the RX 9070 runner and compare the captured failure or crash data with runs on the RX 6800. Done means explaining the intermittent PSO failures or narrowing them to a reproducible race.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ci-cd, computer-graphics, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.