llvm / llvm/offload-test-suite
windows-amd-dxc-d3d12 fails ~50% of the time in post-commit run with 'Failed to create PSO' across many tests
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 18
- Forks
- 39
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 40
Description
`windows-amd-dxc-d3d12` has been failing roughly half of post-commit runs. 16 of the last 30 scheduled runs failed. When we manually re-ran the workflow 8 times against a single SHA to characterize the flakiness, each run produced a different set of failing tests — so this is a race, not a regression.
In the post-commit pipeline, all failures look the same:
```
gpu-exec: error: Failed to create PSO
```
Exit 1, no crash signature in the captured logs. The pre-commit pipeline has separately been seeing actual crashes originating from the same `CreatePSO` call. We don't yet have an explanation for why the two pipelines surface differently.
Runner: RX 9070 / driver 32.0.31007.1017 (from the dxdiag artifact).
Tests run in parallel via lit, each in its own `offloader.exe` process. The root cause is still unclear — it's possible there's a race condition in the AMD UMD around `CreateComputePipelineState`.
Next steps I'd like to take:
1. Enable WER LocalDumps on the runner so the next pre-commit crash captures a user-mode dump. This needs a remote/admin session — it can't be done from a workflow.
2. Get an RX 9070 + matching driver into a dev box for local investigation — can't reproduce on RX 6800 / 32.0.21043.10005, and having admin + free re-run cycles would unblock most things.
3. Longer-term: kernel-mode debugging if the user-mode dump isn't enough.
Per-test frequency and per-run timeline in a comment below.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the post-commit and pre-commit runs using the per-test frequency and timeline from the issue comment, then inspect the gpu-exec CreatePSO failures across the lit-launched offloader.exe processes. Enable WER LocalDumps on the RX 9070 runner and compare the captured failure or crash data with runs on the RX 6800. Done means explaining the intermittent PSO failures or narrowing them to a reproducible race.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- ci-cd, computer-graphics, operating-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100