[GSD-13025] Sysman: Events: Active event keeps getting triggered

[tkatila]

Pre-submission Checklist

• I am using the latest GPU driver version (releases)
• I have searched for similar issues and found none

GPU Hardware

Intel Arc B580

DRI Devices Information

n/a

GPU Detailed Information (lspci output)

n/a

Driver Version

26.22.38646.6

Installed GPU Driver Packages

n/a

Driver Installation Details

n/a

Linux Distribution

Ubuntu 24.04 LTS

Other Linux Distribution

No response

Kernel Version & Boot Parameters

n/a

Actual Behavior

With a custom build of the release (with #950 fixed), when an application is listening for events and there is a reset-requiring event (device wedged or device in survivability mode), call to get any new events returns immediately with the survivability/wedged event. For an application monitoring the system, this behavior is not very optimal as it will most likely block events coming from other devices.

The default zello_sysman doesn't support listening for events more than once, so one has to modify the source code slightly:

diff --git a/level_zero/tools/test/black_box_tests/zello_sysman.cpp b/level_zero/tools/test/black_box_tests/zello_sysman.cpp
index 2ecf2ecf40..ead8a54839 100644
--- a/level_zero/tools/test/black_box_tests/zello_sysman.cpp
+++ b/level_zero/tools/test/black_box_tests/zello_sysman.cpp
@@ -2590,6 +2590,7 @@ int main(int argc, char *argv[]) {
                                                     ZES_EVENT_TYPE_FLAG_DEVICE_ATTACH | ZES_EVENT_TYPE_FLAG_RAS_CORRECTABLE_ERRORS |
                                                     ZES_EVENT_TYPE_FLAG_RAS_UNCORRECTABLE_ERRORS | ZES_EVENT_TYPE_FLAG_FABRIC_PORT_HEALTH | ZES_EVENT_TYPE_FLAG_MEM_HEALTH | ZES_EVENT_TYPE_FLAG_SURVIVABILITY_MODE_DETECTED));
         });
+        while (true)
         testSysmanListenEventsEx(driver, devices,
                                  ZES_EVENT_TYPE_FLAG_DEVICE_RESET_REQUIRED | ZES_EVENT_TYPE_FLAG_DEVICE_DETACH |
                                      ZES_EVENT_TYPE_FLAG_DEVICE_ATTACH | ZES_EVENT_TYPE_FLAG_RAS_CORRECTABLE_ERRORS |

So that it continues listening for events after the initial one.

Expected Behavior

After the initial event, the next event should arrive when it actually arrives. The initial event shouldn't be re-triggered immediately.

Reproduction Rate

Always reproduces - 100%

Steps to Reproduce

• Compile the zello_sysman tool with the "keep listening loop"
• Run zello_sysman with -E
  • Something like sudo -E ZELLO_SYSMAN_USE_ZESINIT=1 ./zello_sysman -E
• Trigger an error that causes device to go into survivability mode (in another terminal)

echo 100 | sudo tee /sys/kernel/debug/dri/<bdf>/inject_csc_hw_error/probability
echo 1 | sudo tee /sys/kernel/debug/dri/<bdf>/inject_csc_hw_error/times

Inject support requires fairly new kernel. Mine is 7.1 with the CONFIG_FAULT_INJECTION=y and CONFIG_FAULT_INJECTION_DEBUG_FS=y enabled in the kernel.

• Observe the zello_sysman app

Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
Device 0 got SURVIVABILITY_MODE_DETECTED event
...

Until ctrl+c is pressed.

Is this a regression?

• Yes, this is a regression - functionality that previously worked is now broken

Last Known Working Driver Version

No response

First Known Failing Driver Version

No response

API Call Logs

No response

strace Logs

No response

System Logs / dmesg Output

No response

Backtrace (if crash or hang occurred)

No response

Source Code / Reproducer

diff --git a/level_zero/tools/test/black_box_tests/zello_sysman.cpp b/level_zero/tools/test/black_box_tests/zello_sysman.cpp
index 2ecf2ecf40..ead8a54839 100644
--- a/level_zero/tools/test/black_box_tests/zello_sysman.cpp
+++ b/level_zero/tools/test/black_box_tests/zello_sysman.cpp
@@ -2590,6 +2590,7 @@ int main(int argc, char *argv[]) {
                                                     ZES_EVENT_TYPE_FLAG_DEVICE_ATTACH | ZES_EVENT_TYPE_FLAG_RAS_CORRECTABLE_ERRORS |
                                                     ZES_EVENT_TYPE_FLAG_RAS_UNCORRECTABLE_ERRORS | ZES_EVENT_TYPE_FLAG_FABRIC_PORT_HEALTH | ZES_EVENT_TYPE_FLAG_MEM_HEALTH | ZES_EVENT_TYPE_FLAG_SURVIVABILITY_MODE_DETECTED));
         });
+        while (true)
         testSysmanListenEventsEx(driver, devices,
                                  ZES_EVENT_TYPE_FLAG_DEVICE_RESET_REQUIRED | ZES_EVENT_TYPE_FLAG_DEVICE_DETACH |
                                      ZES_EVENT_TYPE_FLAG_DEVICE_ATTACH | ZES_EVENT_TYPE_FLAG_RAS_CORRECTABLE_ERRORS |

Command Line / Application Details

No response

oneAPI Version (if applicable)

No response

Screenshots / Video

No response

Additional Notes

No response

[kgibala]

Hi @tkatila,

Thank you for your contribution and for bringing this to our attention.

The issue has been reviewed and addressed. For any further scope or follow-up actions, this has been transferred to the Sysman Team, who will be handling it going forward.

We will provide an update here as soon as there is meaningful progress to share.

Thank you for your patience and understanding.