NVIDIA / NVIDIA/cuda-python

cuda.core: should sticky CUDA errors derive from BaseException?

Open
#2,761 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

cuda.core RFC
Dominant language
Cython
Stars
3.4k
Forks
329
Avg merge
1d 23h
Merged PRs (30d)
116

Description

Question

Should cuda.core raise sticky CUDA errors (device faults after which every further CUDA call returns the same error) as a dedicated exception type that derives from BaseException rather than Exception?

This came up in the review of #2759 (see the thread on docs/source/error_handling.rst, "Sticky errors"). Today all driver and runtime errors are raised as cuda.core.CUDAError, which derives from Exception. #2759 documents that sticky errors are raised like any other error and that the application decides how to shut down.

Motivation

After a sticky fault the CUDA documentation calls for the process to be terminated and relaunched; there is no legitimate recovery. A BaseException subclass is not caught by except Exception:, so naive "if anything goes wrong, retry" code would not swallow the fault and keep issuing CUDA work that is guaranteed to fail. It remains catchable for code that explicitly wants to (except BaseException, or the class itself). KeyboardInterrupt and SystemExit are the stdlib precedent for exit-like conditions.

What it would change

  • A new public exception type beside CUDAError. It cannot derive from CUDAError, because that would make it an Exception. Consequently except CUDAError: stops catching sticky faults, which is a visible behavior change for existing users.
  • _check_driver_error / _check_runtime_error branch on a frozen set of codes. Derived from the shipped error explanations ("any further CUDA work will return the same error"), the driver set is CUDA_ERROR_CONTAINED (226), ILLEGAL_ADDRESS (700), LAUNCH_TIMEOUT (702), HARDWARE_STACK_ERROR (714), ILLEGAL_INSTRUCTION (715), MISALIGNED_ADDRESS (716), INVALID_ADDRESS_SPACE (717), INVALID_PC (718), LAUNCH_FAILED (719), TENSOR_MEMORY_LEAK (721), MPS_CLIENT_TERMINATED (810), EXTERNAL_DEVICE (911). The runtime table has the same codes. CUDA_ERROR_ASSERT is documented as context-fatal, not process-fatal, and would be excluded. A test can cross-check the set against the explanation text.
  • Internal except Exception: sites need review, since a BaseException skips them: the Buffer deallocation callback, the free-then-reraise blocks in _memoryview.pyx, the virtual-memory rollback closure, the capture-tail chaining in GraphBuilder. Bare except: sites are unaffected.
  • Tests for a real sticky fault must run in a subprocess, because the fault poisons the test process.
  • Docs (error_handling.rst, api.rst), __all__, stubs, release notes.

Considerations

  • PEP 352 says user-defined exceptions should derive from Exception; direct BaseException subclasses are reserved for conditions that should not be caught by default. A process-fatal device fault arguably qualifies, but this is an unusual stance for a library.
  • No ecosystem precedent: CuPy, PyTorch and Numba raise Exception subclasses (RuntimeError or their own) for the same driver codes. In mixed stacks the other library's exception type is what a user sees first anyway.
  • pytest, concurrent.futures and threading.excepthook treat a custom BaseException as an ordinary failure. asyncio propagates it out of the event loop, cancelling other tasks, which is the intended "stop" but abrupt.

Alternative

Keep sticky errors as CUDAError (so existing handlers keep working) but raise a dedicated subclass, or set a sticky attribute, so code can recognize them without string matching. This gives the machine-readable signal without the except Exception speed bump.

Decision requested

  1. Do we want the BaseException speed bump at the cost of except CUDAError: no longer catching sticky faults?
  2. If not, is a CUDAError subclass (or attribute) for sticky faults worth adding?

Refs: #2758 (error-handling policy RFC), #2759 (implementation and review thread).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read docs/source/error_handling.rst and the #2759 review thread for the current sticky-error policy, then inspect _check_driver_error, _check_runtime_error, and the listed internal exception handlers. Resolve the exception-hierarchy and sticky-code decision before covering subprocess tests, API exports, stubs, documentation, and release notes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, documentation, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.