ContainerRuntimeCheck: containerd plugin fallback in _gpu_operator_installed is unreachable

Summary

ContainerRuntimeCheck._gpu_operator_installed (isvtest/src/isvtest/validations/host.py) never executes its ctr plugins ls fallback. Hosts that register the NVIDIA runtime only as a containerd plugin — with no matching file under /etc/containerd/ — are reported as nvidia-container-runtime not installed.

Detail

The probe runs:

grep -rl 'nvidia' /etc/containerd/ 2>/dev/null | head -1 ||
ctr plugins ls 2>/dev/null | grep -i nvidia | head -1 ||
echo '__not_configured__'

|| binds looser than |, so this is three pipelines. A pipeline's exit status is its last command's, and head -1 exits 0 on empty input — so the first pipeline always succeeds and neither the ctr plugins ls fallback nor the __not_configured__ sentinel ever runs.

Reproduced against a directory with no matching file:

grep -rl 'nvidia' <dir> 2>/dev/null | head -1   ->  exit=0, empty output
full expression                                 ->  empty output (fallback never reached)

config_out comes back empty, so the method returns False.

Impact

The docstring states the check verifies the runtime is "referenced in the containerd configuration or registered as a containerd plugin". The second condition has never been evaluated. Detection levels 3–5 (containerd / runc / crun + nvidia-container-runtime) fail on correctly configured hosts — the hosts #580 was written to support.

Suggested fix

Collect from both sources, then test the captured output rather than the pipeline status:

{ grep -rl 'nvidia' /etc/containerd/ 2>/dev/null;
  ctr plugins ls 2>/dev/null | grep -i nvidia; } | head -1 | grep . || echo '__not_configured__'

grep . exits non-zero on empty input, so the sentinel fires correctly. Verified across four cases: config file only, plugin only, neither, and missing /etc/containerd/.

Notes

• Introduced in #580, present on main.
• Also present on releases/0.7.x via the backport in #583, where it was deliberately left unchanged to preserve main-first ordering — see https://github.com/NVIDIA/ai-cloud-validation/pull/583#discussion_r3792676738.
• Fix main first, then cherry-pick to releases/0.7.x if a 0.7.x release still needs it.
• Originally surfaced by CodeRabbit on #583.

Start in isvtest/src/isvtest/validations/host.py at ContainerRuntimeCheck._gpu_operator_installed, then reproduce the probe with config-only, plugin-only, neither, and missing-directory cases. Check that the containerd plugin fallback is evaluated and that the method distinguishes configured hosts from the not_configured sentinel.