intel / intel/compute-runtime

[GSD-13025] Sysman: Events: Active event keeps getting triggered

Open
#951 1 comment 0 reactions 0 assignees View on GitHub
Component: Level Zero Sysman OS: Linux Type: Bug
Dominant language
C++
Stars
1.4k
Forks
300
PR merge metrics
No merged PRs in 30d

Description

### Pre-submission Checklist

- [x] I am using the latest GPU driver version ([releases](https://github.com/intel/compute-runtime/releases))
- [x] I have searched for similar issues and found none

### GPU Hardware

Intel Arc B580

### DRI Devices Information

n/a

### GPU Detailed Information (lspci output)

n/a

### Driver Version

26.22.38646.6

### Installed GPU Driver Packages

n/a

### Driver Installation Details

n/a

### Linux Distribution

Ubuntu 24.04 LTS

### Other Linux Distribution

_No response_

### Kernel Version & Boot Parameters

n/a

### Actual Behavior

With a custom build of the release (with #950 fixed), when an application is listening for events and there is a reset-requiring event (device wedged or device in survivability mode), call to get any new events returns immediately with the survivability/wedged event. For an application monitoring the system, this behavior is not very optimal as it will most likely block events coming from other devices.

The default zello_sysman doesn't support listening for events more than once, so one has to modify the source code slightly:
```
diff --git a/level_zero/tools/test/black_box_tests/zello_sysman.cpp b/level_zero/tools/test/black_box_tests/zello_sysman.cpp
index 2ecf2ecf40..ead8a54839 100644
--- a/level_zero/tools/test/black_box_tests/zello_sysman.cpp
+++ b/level_zero/tools/test/black_box_tests/zello_sysman.cpp
@@ -2590,6 +2590,7 @@ int main(int argc, char *argv[]) {
ZES_EVENT_TYPE_FLAG_DEVICE_ATTACH | ZES_EVENT_TYPE_FLAG_RAS_CORRECTABLE_ERRORS |
ZES_EVENT_TYPE_FLAG_RAS_UNCORRECTABLE_ERRORS | ZES_EVENT_TYPE_FLAG_FABRIC_PORT_HEALTH | ZES_EVENT_TYPE_FLAG_MEM_HEALTH | ZES_EVENT_TYPE_FLAG_SURVIVABILITY_MODE_DETECTED));
});
+ while (true)
testSysmanListenEventsEx(driver, devices,
ZES_EVENT_TYPE_FLAG_DEVICE_RESET_REQUIRED | ZES_EVENT_TYPE_FLAG_DEVICE_DETACH |
ZES_EVENT_TYPE_FLAG_DEVICE_ATTACH | ZES_EVENT_TYPE_FLAG_RAS_CORRECTABLE_ERRORS |
```
So that it continues listening for events after the initial one.

### Expected Behavior

After the initial event, the next event should arrive when it actually arrives. The initial event shouldn't be re-triggered immediately.

### Reproduction Rate

Always reproduces - 100%

### Steps to Reproduce

* Compile the zello_sysman tool with the "keep listening loop"
* Run zello_sysman with `-E`
* Something like `sudo -E ZELLO_SYSMAN_USE_ZESINIT=1 ./zello_sysman -E`
* Trigger an error that causes device to go into survivability mode (in another terminal)
```
echo 100 | sudo tee /sys/kernel/debug/dri//inject_csc_hw_error/probability
echo 1 | sudo tee /sys/kernel/debug/dri//inject_csc_hw_error/times
```
Inject support requires fairly new kernel. Mine is 7.1 with the `CONFIG_FAULT_INJECTION=y` and `CONFIG_FAULT_INJECTION_DEBUG_FS=y` enabled in the kernel.
* Observe the zello_sysman app

```
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
...
```
Until ctrl+c is pressed.

### Is this a regression?

- [ ] Yes, this is a regression - functionality that previously worked is now broken

### Last Known Working Driver Version

_No response_

### First Known Failing Driver Version

_No response_

### API Call Logs

_No response_

### strace Logs

_No response_

### System Logs / dmesg Output

_No response_

### Backtrace (if crash or hang occurred)

_No response_

### Source Code / Reproducer

```
diff --git a/level_zero/tools/test/black_box_tests/zello_sysman.cpp b/level_zero/tools/test/black_box_tests/zello_sysman.cpp
index 2ecf2ecf40..ead8a54839 100644
--- a/level_zero/tools/test/black_box_tests/zello_sysman.cpp
+++ b/level_zero/tools/test/black_box_tests/zello_sysman.cpp
@@ -2590,6 +2590,7 @@ int main(int argc, char *argv[]) {
ZES_EVENT_TYPE_FLAG_DEVICE_ATTACH | ZES_EVENT_TYPE_FLAG_RAS_CORRECTABLE_ERRORS |
ZES_EVENT_TYPE_FLAG_RAS_UNCORRECTABLE_ERRORS | ZES_EVENT_TYPE_FLAG_FABRIC_PORT_HEALTH | ZES_EVENT_TYPE_FLAG_MEM_HEALTH | ZES_EVENT_TYPE_FLAG_SURVIVABILITY_MODE_DETECTED));
});
+ while (true)
testSysmanListenEventsEx(driver, devices,
ZES_EVENT_TYPE_FLAG_DEVICE_RESET_REQUIRED | ZES_EVENT_TYPE_FLAG_DEVICE_DETACH |
ZES_EVENT_TYPE_FLAG_DEVICE_ATTACH | ZES_EVENT_TYPE_FLAG_RAS_CORRECTABLE_ERRORS |
```

### Command Line / Application Details

_No response_

### oneAPI Version (if applicable)

_No response_

### Screenshots / Video

_No response_

### Additional Notes

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with level_zero/tools/test/black_box_tests/zello_sysman.cpp and run zello_sysman -E using the keep-listening loop described in the issue. Reproduce the survivability-mode event with the debugfs fault-injection commands, then trace the event-listening path to identify why the same event is returned repeatedly. Done means the initial event is reported once and later calls wait for a genuinely new event.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux, ubuntu
Domain
operating-systems, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.