oneapi-src / oneapi-src/unified-runtime

Loader corruption when running CTS with native cpu + level zero enabled

Open
#2,511 2 comments 0 reactions 1 assignee View on GitHub

@RossBrunton is already working on this.

Since Jan 17, 2025.

loader
Dominant language
C++
Stars
57
Forks
120
Avg merge
1d 14h
Merged PRs (30d)
1

Description

This issue only applies to the branch for https://github.com/oneapi-src/unified-runtime/pull/2479. I'm recording it here because it doesn't affect how our CI currently works so it isn't necessarily a blocker to merging.

To reproduce check out a branch containing the changes for https://github.com/oneapi-src/unified-runtime/pull/2479, enable and build the L0 and native cpu adapters and run the test-enqueue cts suite. The problem is intermittent but it shouldn't take many attempts to see either a segfault in, or an error returned from urQueueCreate.

The pathology of this behaviour is that the context handle (importantly it seems to be the loader handle, not the adapter handle) passed to urQueueCreate is corrupted somehow, resulting in the wrong adapter's implementation of urQueueCreate getting called. Most commonly this happens during a native cpu test, where the level zero implementation is called and returns UR_RESULT_ERROR_INVALID_DEVICE when it doesn't recognize the (native cpu) device. The problem doesn't respond well to debuggers but I've instrumented various bits of loader and adapter code and been able to observe the address for the urQueueCreate entry point changing from test to test when this occurs.

This same issue is behind various other spooky behaviours in a few test suites. You can see problems running the test-queue suite, and sometimes rather than what's described above in test-enqueue you'll get wrong results or a hang.

Removing this line from the l0 urContextRelease implementation (effectively leaking all the contexts) makes the problem go away

No problems are observed when only the native cpu + opencl adapters are enabled, strangely the opencl adapter seems completely unaffected.

The issue isn't anything to do with a bad urEnqueue operation (initially I thought it might be related to a bad buffer operation or something). It can be reproduced in the test-queue suite running tests that only call the following entry points:

   ---> urAdapterGet
   ---> urAdapterGetInfo
   ---> urContextCreate
   ---> urContextRelease
   ---> urDeviceGet
   ---> urDeviceGetInfo
   ---> urPlatformGet
   ---> urPlatformGetInfo
   ---> urQueueCreate
   ---> urQueueGetInfo
   ---> urQueueRelease

Valgrind, UB sanitizer and address sanitizer have all come up empty handed, although this must be some kind of memory corruption. As mentioned it doesn't reproduce while running in a debugger for the most part so it isn't too surprising that these tools are enough to mess with whatever's going on.

My current best guess is that something in the l0 adapter is retaining a reference to a data member from a context after it gets destroyed, and that's getting used somewhere such that bad memory accesses occur, although I haven't actually produced any evidence of this.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.