cublas status execution failed error

edit: this doesn't appear if use_turbo is unchecked

in the run alphafold section of the advanced notebook, this error appears while the first model is being run:

---

UnfilteredStackTrace Traceback (most recent call last)

in ()
206 # predict
--> 207 prediction_result, (r, t) = cf.to(model_runner.predict(processed_feature_dict, random_seed=seed),"cpu")
208

11 frames

UnfilteredStackTrace: RuntimeError: Internal: CUBLAS_STATUS_EXECUTION_FAILED

The stack trace below excludes JAX-internal frames.
The preceding is the original exception that occurred, unmodified.

---

The above exception was the direct cause of the following exception:

RuntimeError Traceback (most recent call last)

/usr/local/lib/python3.7/dist-packages/jax/interpreters/xla.py in _execute_compiled(compiled, avals, handlers, kept_var_idx, *args)
910 for i, x in enumerate(args)
911 if x is not token and i in kept_var_idx))
--> 912 out_bufs = compiled.execute(input_bufs)
913 check_special(xla_call_p.name, out_bufs)
914 return [handler(*bs) for handler, bs in zip(handlers, _partition_outputs(avals, out_bufs))]

RuntimeError: Internal: CUBLAS_STATUS_EXECUTION_FAILED

Start with the advanced notebook's run alphafold section and reproduce the first-model run with use_turbo checked. Read the traceback around model_runner.predict and the reported CUBLAS_STATUS_EXECUTION_FAILED; done means the first model completes without this error.