[GSD-13025] Sysman: Events: Active event keeps getting triggered
- Dominant language
- C++
- Stars
- 1.4k
- Forks
- 300
- PR merge metrics
- No merged PRs in 30d
Description
### Pre-submission Checklist
- [x] I am using the latest GPU driver version ([releases](https://github.com/intel/compute-runtime/releases))
- [x] I have searched for similar issues and found none
### GPU Hardware
Intel Arc B580
### DRI Devices Information
n/a
### GPU Detailed Information (lspci output)
n/a
### Driver Version
26.22.38646.6
### Installed GPU Driver Packages
n/a
### Driver Installation Details
n/a
### Linux Distribution
Ubuntu 24.04 LTS
### Other Linux Distribution
_No response_
### Kernel Version & Boot Parameters
n/a
### Actual Behavior
With a custom build of the release (with #950 fixed), when an application is listening for events and there is a reset-requiring event (device wedged or device in survivability mode), call to get any new events returns immediately with the survivability/wedged event. For an application monitoring the system, this behavior is not very optimal as it will most likely block events coming from other devices.
The default zello_sysman doesn't support listening for events more than once, so one has to modify the source code slightly:
```
diff --git a/level_zero/tools/test/black_box_tests/zello_sysman.cpp b/level_zero/tools/test/black_box_tests/zello_sysman.cpp
index 2ecf2ecf40..ead8a54839 100644
--- a/level_zero/tools/test/black_box_tests/zello_sysman.cpp
+++ b/level_zero/tools/test/black_box_tests/zello_sysman.cpp
@@ -2590,6 +2590,7 @@ int main(int argc, char *argv[]) {
ZES_EVENT_TYPE_FLAG_DEVICE_ATTACH | ZES_EVENT_TYPE_FLAG_RAS_CORRECTABLE_ERRORS |
ZES_EVENT_TYPE_FLAG_RAS_UNCORRECTABLE_ERRORS | ZES_EVENT_TYPE_FLAG_FABRIC_PORT_HEALTH | ZES_EVENT_TYPE_FLAG_MEM_HEALTH | ZES_EVENT_TYPE_FLAG_SURVIVABILITY_MODE_DETECTED));
});
+ while (true)
testSysmanListenEventsEx(driver, devices,
ZES_EVENT_TYPE_FLAG_DEVICE_RESET_REQUIRED | ZES_EVENT_TYPE_FLAG_DEVICE_DETACH |
ZES_EVENT_TYPE_FLAG_DEVICE_ATTACH | ZES_EVENT_TYPE_FLAG_RAS_CORRECTABLE_ERRORS |
```
So that it continues listening for events after the initial one.
### Expected Behavior
After the initial event, the next event should arrive when it actually arrives. The initial event shouldn't be re-triggered immediately.
### Reproduction Rate
Always reproduces - 100%
### Steps to Reproduce
* Compile the zello_sysman tool with the "keep listening loop"
* Run zello_sysman with `-E`
* Something like `sudo -E ZELLO_SYSMAN_USE_ZESINIT=1 ./zello_sysman -E`
* Trigger an error that causes device to go into survivability mode (in another terminal)
```
echo 100 | sudo tee /sys/kernel/debug/dri//inject_csc_hw_error/probability
echo 1 | sudo tee /sys/kernel/debug/dri//inject_csc_hw_error/times
```
Inject support requires fairly new kernel. Mine is 7.1 with the `CONFIG_FAULT_INJECTION=y` and `CONFIG_FAULT_INJECTION_DEBUG_FS=y` enabled in the kernel.
* Observe the zello_sysman app
```
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
...
```
Until ctrl+c is pressed.
### Is this a regression?
- [ ] Yes, this is a regression - functionality that previously worked is now broken
### Last Known Working Driver Version
_No response_
### First Known Failing Driver Version
_No response_
### API Call Logs
_No response_
### strace Logs
_No response_
### System Logs / dmesg Output
_No response_
### Backtrace (if crash or hang occurred)
_No response_
### Source Code / Reproducer
```
diff --git a/level_zero/tools/test/black_box_tests/zello_sysman.cpp b/level_zero/tools/test/black_box_tests/zello_sysman.cpp
index 2ecf2ecf40..ead8a54839 100644
--- a/level_zero/tools/test/black_box_tests/zello_sysman.cpp
+++ b/level_zero/tools/test/black_box_tests/zello_sysman.cpp
@@ -2590,6 +2590,7 @@ int main(int argc, char *argv[]) {
ZES_EVENT_TYPE_FLAG_DEVICE_ATTACH | ZES_EVENT_TYPE_FLAG_RAS_CORRECTABLE_ERRORS |
ZES_EVENT_TYPE_FLAG_RAS_UNCORRECTABLE_ERRORS | ZES_EVENT_TYPE_FLAG_FABRIC_PORT_HEALTH | ZES_EVENT_TYPE_FLAG_MEM_HEALTH | ZES_EVENT_TYPE_FLAG_SURVIVABILITY_MODE_DETECTED));
});
+ while (true)
testSysmanListenEventsEx(driver, devices,
ZES_EVENT_TYPE_FLAG_DEVICE_RESET_REQUIRED | ZES_EVENT_TYPE_FLAG_DEVICE_DETACH |
ZES_EVENT_TYPE_FLAG_DEVICE_ATTACH | ZES_EVENT_TYPE_FLAG_RAS_CORRECTABLE_ERRORS |
```
### Command Line / Application Details
_No response_
### oneAPI Version (if applicable)
_No response_
### Screenshots / Video
_No response_
### Additional Notes
_No response_
Contributor guide
Research direction
Start with level_zero/tools/test/black_box_tests/zello_sysman.cpp and run zello_sysman -E using the keep-listening loop described in the issue. Reproduce the survivability-mode event with the debugfs fault-injection commands, then trace the event-listening path to identify why the same event is returned repeatedly. Done means the initial event is reported once and later calls wait for a genuinely new event.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, linux, ubuntu
- Domain
- operating-systems, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100