StreamFlags::NON_BLOCKING is unsound because of fringe asynchronous memory copy behavior in CUDA

Streams with NON_BLOCKING exhibit very confusing and very dangerous behavior with regards to memcpy due to odd CUDA semantics, per the driver API docs:

For transfers from pageable host memory to device memory, a stream sync is performed before the copy is initiated. The function will return once the pageable buffer has been copied to the staging memory for DMA transfer to device memory, but the DMA to final destination may not have completed.

Because NON_BLOCKING streams do not synchronize with the null (default) stream, this leads to potential race conditions. NVIDIA appears to be aware of this issue, but in the mean time, it may be beneficial to implicitly disable NON_BLOCKING for now. Especially since cust does not expose stream ordered memory allocation.

This is what appears to be happening in the add example sometimes not doing anything on certain systems.

Read the CUDA driver API's API sync behavior documentation and reproduce the issue with the add example on an affected system. A satisfactory result should establish whether the NON_BLOCKING behavior causes the race and clearly define the resulting stream behavior.