First SYCL/UR queue submission after idle-following-sustained-workload fails; not reproduced via raw Level Zero (Iris Xe, Windows, oneAPI 2026.1)
- Dominant language
- LLVM
- Stars
- 1.5k
- Forks
- 854
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 137
Description
### Describe the bug
On a single Intel Iris Xe (Tiger Lake) GPU, Windows 11, driver 32.0.101.7088, oneAPI 2026.1: after ~2 minutes of continuous queue activity followed by several minutes idle, the first `zeCommandQueueExecuteCommandLists` after the idle window fails with `ZE_RESULT_ERROR_UNKNOWN` at the L0 trace level, surfacing through SYCL/UR as `UR_RESULT_ERROR_DEVICE_LOST` (minimal reproducer below) or `UR_RESULT_ERROR_UNKNOWN` (lc0, a chess engine using `sycl-fp16`). Idle alone (no prior sustained activity) does not trigger it; the same shape reimplemented directly against raw Level Zero (no SYCL/UR) does not reproduce it either. A periodic 1-byte keep-alive submission every 60s during idle reliably avoids it. Full diagnostic trail below.
### To reproduce
```
set ONEAPI_DEVICE_SELECTOR=level_zero:0
icpx -fsycl -fexceptions idle_queue_resume.cc -o idle_queue_resume.exe
idle_queue_resume.exe
```
Runs a GPU queue: ~140s of sustained `memset` submissions, then idles 260s, then submits once more. The final submission fails. Threshold observed: idle 60s clean, idle 260s fails reliably (2/2-5/5 across variants below). `RESUBMIT_MODE` (env var: `same`/`newqueue`/`newcontext`), `PHASE1_SECONDS`, `IDLE_SECONDS`, `KEEPWARM_SECONDS` control the variants discussed below.
idle_queue_resume.cc (SYCL reproducer)
```cpp
// Minimal standalone reproducer: first Level-Zero command-queue submission
// after a long idle window fails with ZE_RESULT_ERROR_UNKNOWN.
//
// Build (oneAPI 2026.1, Windows): icx -fsycl idle_queue_resume.cc -o idle_queue_resume.exe
// Run: idle_queue_resume.exe
//
// Observed (Intel Iris Xe, driver 32.0.101.7088, oneAPI 2026.1):
// phase 1 (140s of queue activity): OK
// phase 2 (260s idle): -
// phase 3 (first submission after idle): throws
// "level_zero backend failed with error: 2147483646 (UR_RESULT_ERROR_UNKNOWN)"
// UR_L0_DEBUG=1 shows the failing call:
// Error (ZE_RESULT_ERROR_UNKNOWN) in zeCommandQueueExecuteCommandLists
// The same program with a trivial submission every 60s during the idle
// phase completes cleanly.
#include
#include
#include
#include
#include
#include
static double SecondsSince(std::chrono::steady_clock::time_point t0) {
return std::chrono::duration(std::chrono::steady_clock::now() - t0)
.count();
}
int main() {
sycl::queue q{sycl::gpu_selector_v, sycl::property::queue::in_order()};
const auto dev = q.get_device();
const auto plat = dev.get_platform();
std::printf("device: %s\n",
dev.get_info().c_str());
std::printf("platform: %s\n",
plat.get_info().c_str());
std::printf("backend: %s\n",
q.get_backend() == sycl::backend::ext_oneapi_level_zero
? "level_zero" : "other (not level_zero -- pin with "
"ONEAPI_DEVICE_SELECTOR=level_zero:0)");
std::printf("driver_version: %s\n",
dev.get_info().c_str());
constexpr size_t kBytes = 4 * 1024 * 1024;
int* buf = sycl::malloc_device(kBytes / sizeof(int), q);
if (!buf) {
std::printf("malloc_device failed\n");
return 2;
}
const int idle_seconds = std::getenv("IDLE_SECONDS")
? std::atoi(std::getenv("IDLE_SECONDS"))
: 260;
const int keepwarm_seconds = std::getenv("KEEPWARM_SECONDS")
? std::atoi(std::getenv("KEEPWARM_SECONDS"))
: 0;
// Make the warm-up duration configurable so idle-duration-alone can be
// tested separately from idle-after-prior-workload. PHASE1_SECONDS=0
// skips phase 1 entirely (queue created and never submitted to before
// idling).
const int phase1_seconds = std::getenv("PHASE1_SECONDS")
? std::atoi(std::getenv("PHASE1_SECONDS"))
: 140;
// Phase 1: sustained queue activity, mirroring a long engine search.
const auto t0 = std::chrono::steady_clock::now();
int it = 0;
if (phase1_seconds > 0) {
while (SecondsSince(t0) < phase1_seconds) {
q.memset(buf, it & 0xff, kBytes);
if (++it % 20 == 0) q.wait_and_throw();
}
q.wait_and_throw();
}
std::printf("phase 1 done: %d submissions in %.1fs (requested %ds)\n", it,
SecondsSince(t0), phase1_seconds);
// Phase 2: idle, optionally pinging to keep the queue warm.
std::printf("idling %ds (keepwarm every %ds)...\n", idle_seconds,
keepwarm_seconds);
const auto idle_start = std::chrono::steady_clock::now();
auto last_ping = idle_start;
while (SecondsSince(idle_start) < idle_seconds) {
std::this_thread::sleep_for(std::chrono::milliseconds(200));
if (keepwarm_seconds > 0 &&
SecondsSince(last_ping) >= keepwarm_seconds) {
q.memset(buf, 0, 1).wait_and_throw();
last_ping = std::chrono::steady_clock::now();
std::printf(" keepwarm ping at %.0fs idle\n", SecondsSince(idle_start));
}
}
// Phase 3: the first submission after the idle window. Resubmit on the
// original queue, on a new queue sharing the original context/device, or
// on a queue backed by a brand-new context -- diagnostic data points on
// where the failing state lives, not a claim that any of these is a safe
// recovery.
const std::string resubmit_mode = std::getenv("RESUBMIT_MODE")
? std::getenv("RESUBMIT_MODE")
: "same";
std::printf("resubmit_mode: %s\n", resubmit_mode.c_str());
try {
if (resubmit_mode == "same") {
q.memset(buf, 7, kBytes).wait_and_throw();
} else if (resubmit_mode == "newqueue") {
// New queue, same context and device as the one that just idled.
sycl::queue q2{q.get_context(), dev, sycl::property::queue::in_order()};
q2.memset(buf, 7, kBytes).wait_and_throw();
} else if (resubmit_mode == "newcontext") {
// New queue with an implicitly new context -- allocation must be
// against the new context too, so this does not reuse `buf`.
sycl::queue q3{dev, sycl::property::queue::in_order()};
int* buf3 = sycl::malloc_device(kBytes / sizeof(int), q3);
if (!buf3) {
std::printf("phase 3 (newcontext): malloc_device failed\n");
return 2;
}
q3.memset(buf3, 7, kBytes).wait_and_throw();
sycl::free(buf3, q3);
} else {
std::printf("unknown RESUBMIT_MODE: %s\n", resubmit_mode.c_str());
return 3;
}
std::printf("phase 3 (%s): resumed OK\n", resubmit_mode.c_str());
} catch (const sycl::exception& e) {
std::printf("phase 3 (%s) FAILED: %s\n", resubmit_mode.c_str(), e.what());
return 1;
}
sycl::free(buf, q);
return 0;
}
```
idle_queue_resume_native_l0.cc (raw Level Zero reproducer, no SYCL/UR)
```cpp
// Native Level Zero reproducer: bypasses SYCL and the Unified Runtime
// adapter entirely, calling the L0 driver API directly, to test whether
// the idle-resume failure originates in the SYCL/UR layer or below it.
// Mirrors idle_queue_resume.cc's shape (create queue, sustained workload,
// idle, resubmit) and reproduces the exact call sequence seen in the
// original UR_L0_DEBUG trace: command list close -> queue execute -> (on
// failure) fence reset -> list reset.
//
// Build (oneAPI 2026.1 + Level Zero SDK, from a vcvars64 + setvars shell):
// cl /std:c++17 /EHsc /I"C:\Program Files\LevelZeroSDK\1.24.0\include" ^
// idle_queue_resume_native_l0.cc /link ^
// /LIBPATH:"C:\Program Files\LevelZeroSDK\1.24.0\lib" ze_loader.lib
// Run:
// idle_queue_resume_native_l0.exe
// Env: IDLE_SECONDS (default 260), PHASE1_SECONDS (default 140, 0 skips
// the warm-up workload).
#include
#include
#include
#include
#include
#include
#include
static double SecondsSince(std::chrono::steady_clock::time_point t0) {
return std::chrono::duration(std::chrono::steady_clock::now() - t0)
.count();
}
#define ZE_CHECK(call) \
do { \
ze_result_t _r = (call); \
if (_r != ZE_RESULT_SUCCESS) { \
std::printf("FAILED %s -> 0x%x\n", #call, (unsigned)_r); \
return (int)_r; \
} \
} while (0)
int main() {
ZE_CHECK(zeInit(0));
uint32_t driver_count = 0;
ZE_CHECK(zeDriverGet(&driver_count, nullptr));
std::vector drivers(driver_count);
ZE_CHECK(zeDriverGet(&driver_count, drivers.data()));
ze_device_handle_t device = nullptr;
ze_driver_handle_t driver = nullptr;
for (auto d : drivers) {
uint32_t device_count = 0;
zeDeviceGet(d, &device_count, nullptr);
std::vector devices(device_count);
zeDeviceGet(d, &device_count, devices.data());
for (auto dev : devices) {
ze_device_properties_t props{ZE_STRUCTURE_TYPE_DEVICE_PROPERTIES};
zeDeviceGetProperties(dev, &props);
if (props.type == ZE_DEVICE_TYPE_GPU) {
device = dev;
driver = d;
std::printf("device: %s\n", props.name);
break;
}
}
if (device) break;
}
if (!device) {
std::printf("no GPU device found\n");
return 2;
}
ze_driver_properties_t drv_props{ZE_STRUCTURE_TYPE_DRIVER_PROPERTIES};
zeDriverGetProperties(driver, &drv_props);
std::printf("driver_version: 0x%x\n", drv_props.driverVersion);
std::printf("backend: native level_zero (no SYCL/UR)\n");
ze_context_desc_t ctx_desc{ZE_STRUCTURE_TYPE_CONTEXT_DESC};
ze_context_handle_t context = nullptr;
ZE_CHECK(zeContextCreate(driver, &ctx_desc, &context));
uint32_t qgroup_count = 0;
zeDeviceGetCommandQueueGroupProperties(device, &qgroup_count, nullptr);
std::vector qgroups(qgroup_count);
for (auto& g : qgroups)
g.stype = ZE_STRUCTURE_TYPE_COMMAND_QUEUE_GROUP_PROPERTIES;
zeDeviceGetCommandQueueGroupProperties(device, &qgroup_count,
qgroups.data());
uint32_t ordinal = 0;
for (uint32_t i = 0; i < qgroup_count; ++i) {
if (qgroups[i].flags & ZE_COMMAND_QUEUE_GROUP_PROPERTY_FLAG_COMPUTE) {
ordinal = i;
break;
}
}
ze_command_queue_desc_t q_desc{ZE_STRUCTURE_TYPE_COMMAND_QUEUE_DESC};
q_desc.ordinal = ordinal;
q_desc.index = 0;
q_desc.mode = ZE_COMMAND_QUEUE_MODE_DEFAULT;
ze_command_queue_handle_t queue = nullptr;
ZE_CHECK(zeCommandQueueCreate(context, device, &q_desc, &queue));
ze_command_list_desc_t cl_desc{ZE_STRUCTURE_TYPE_COMMAND_LIST_DESC};
cl_desc.commandQueueGroupOrdinal = ordinal;
ze_command_list_handle_t cmdlist = nullptr;
ZE_CHECK(zeCommandListCreate(context, device, &cl_desc, &cmdlist));
ze_fence_desc_t fence_desc{ZE_STRUCTURE_TYPE_FENCE_DESC};
ze_fence_handle_t fence = nullptr;
ZE_CHECK(zeFenceCreate(queue, &fence_desc, &fence));
constexpr size_t kBytes = 4 * 1024 * 1024;
ze_device_mem_alloc_desc_t mem_desc{ZE_STRUCTURE_TYPE_DEVICE_MEM_ALLOC_DESC};
void* buf = nullptr;
ZE_CHECK(zeMemAllocDevice(context, &mem_desc, kBytes, 1, device, &buf));
auto submit_fill = [&](uint8_t pattern) -> ze_result_t {
ze_result_t r = zeCommandListAppendMemoryFill(
cmdlist, buf, &pattern, 1, kBytes, nullptr, 0, nullptr);
if (r != ZE_RESULT_SUCCESS) return r;
r = zeCommandListClose(cmdlist);
if (r != ZE_RESULT_SUCCESS) return r;
r = zeCommandQueueExecuteCommandLists(queue, 1, &cmdlist, fence);
if (r != ZE_RESULT_SUCCESS) return r;
r = zeFenceHostSynchronize(fence, UINT64_MAX);
if (r != ZE_RESULT_SUCCESS) return r;
zeFenceReset(fence);
zeCommandListReset(cmdlist);
return ZE_RESULT_SUCCESS;
};
const int idle_seconds = std::getenv("IDLE_SECONDS")
? std::atoi(std::getenv("IDLE_SECONDS"))
: 260;
const int phase1_seconds = std::getenv("PHASE1_SECONDS")
? std::atoi(std::getenv("PHASE1_SECONDS"))
: 140;
const auto t0 = std::chrono::steady_clock::now();
int it = 0;
if (phase1_seconds > 0) {
while (SecondsSince(t0) < phase1_seconds) {
ze_result_t r = submit_fill((uint8_t)(it & 0xff));
if (r != ZE_RESULT_SUCCESS) {
std::printf("phase 1 FAILED at iter %d: 0x%x\n", it, (unsigned)r);
return 1;
}
++it;
}
}
std::printf("phase 1 done: %d submissions in %.1fs (requested %ds)\n", it,
SecondsSince(t0), phase1_seconds);
std::printf("idling %ds...\n", idle_seconds);
std::this_thread::sleep_for(std::chrono::seconds(idle_seconds));
ze_result_t r = submit_fill(7);
if (r != ZE_RESULT_SUCCESS) {
const char* name = r == ZE_RESULT_ERROR_DEVICE_LOST ? "ZE_RESULT_ERROR_DEVICE_LOST"
: r == ZE_RESULT_ERROR_UNKNOWN ? "ZE_RESULT_ERROR_UNKNOWN"
: "other";
std::printf("phase 3 FAILED: 0x%x (%s)\n", (unsigned)r, name);
return 1;
}
std::printf("phase 3: resumed OK\n");
return 0;
}
```
### Environment
- OS: Windows 11 (10.0.26200, build 26100.1)
- Target device and vendor: Intel(R) Iris(R) Xe Graphics (Tiger Lake, i5-1135G7), driver 32.0.101.7088
- DPC++ version: oneAPI 2026.1 (SYCL 9 / `sycl9.dll`), UR loader 0.12.0
- Dependencies version: `ur_adapter_level_zero{,_v2}.dll` 2026.1, `ze_loader.dll` 1.24.0, `ze_intel_gpu64.dll` 23.20.101.7088; `sycl-ls` shows exactly one `[level_zero:gpu]` device. No TDR/Display event logged in the Windows event log at failure time.
### Additional context
Four diagnostic experiments narrow this down:
- **Idle alone, no prior activity** (`PHASE1_SECONDS=0`, `IDLE_SECONDS=260`): clean 2/2 -- prior sustained activity is required, not idle duration alone.
- **Same shape via raw Level Zero** (`idle_queue_resume_native_l0.cc` above), no SYCL/UR involved: clean 2/2 -- points at the SYCL runtime / UR `level_zero` adapter rather than the L0 driver in isolation. Caveat: the native repro calls `zeFenceHostSynchronize` after every single submission (fully serialized, ~1.2M submit-wait cycles in 140s), while the SYCL repro pipelines ~20 submissions per `wait_and_throw` (~1.7-2.6M submissions in 140s with several outstanding at once) -- not a proven apples-to-apples concurrency match.
- **Resubmit on a new queue** (`RESUBMIT_MODE=newqueue`, same context as the one that idled): fails 2/2, same `UR_RESULT_ERROR_DEVICE_LOST`.
- **Resubmit on a brand-new context** (`RESUBMIT_MODE=newcontext`, never touched before idling, fresh `malloc_device` allocation): fails 2/2, same error -- so the bad state isn't scoped to the queue or context that idled.
Combined, the most specific claim the evidence supports: some process-scoped state in the SYCL runtime or UR `level_zero`/`level_zero_v2` adapter is left bad by sustained-workload-then-idle, not a specific queue/context handle, and probably not the driver/device alone (a separate native-L0 *process* against the same device/driver is unaffected).
Not the same class as intel/compute-runtime#916 or #921 (both are multi-device enumeration/allocation issues under multi-GPU; this is single-device, one context/queue used successfully for minutes, only the first submission after an idle window fails).
**Ask:**
1. Is an idle queue expected to survive arbitrarily long idleness? If there's a known idle timeout/window after which a queue may be invalidated, please document it (and any driver knob to extend it).
2. If this is a bug, a fix would remove the need for a 60s keep-alive submission that applications currently need to stay alive.
Contributor guide
Research direction
Start with idle_queue_resume.cc and idle_queue_resume_native_l0.cc; build and run the SYCL reproducer with ONEAPI_DEVICE_SELECTOR=level_zero:0, then compare it with the native Level Zero run. Vary IDLE_SECONDS, PHASE1_SECONDS, KEEPWARM_SECONDS, and RESUBMIT_MODE while capturing UR_L0_DEBUG output, focusing on zeCommandQueueExecuteCommandLists. Done means locating the responsible layer and documenting a reproducible resolution for the post-idle submission failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100