Step 3 get stuck in forward

Hello,

I am currently trying to run the Deepspeed Chat example for RLHF training but experiencing some problems with Step 3. After running for several hours, the process seems to get stuck during the forward pass in Transformers.

Environment:

Please find the details of the environment in which I am running the process:

Deepspeed Version: 0.9.5
Transformers Version: 4.31.0.dev0
PyTorch Version: 2.0.1+cu117
Python Version: 3.9
GPU model: A100
Model: opt-13B

Expected Behavior:

The process should complete Step 3 of RLHF training without getting stuck and proceed to the subsequent steps.

Actual Behavior:

The process gets stuck during Step 3 of RLHF training. The issue seems to be occurring during the forward pass in the Transformers. This behavior may happens after the process has been running for several hours.

Additional Information:

Upon further investigation, we have found that the function THPVariable_tensor in PyTorch is being called, which seems to be where the process is getting stuck. This function is part of the tensor creation process in PyTorch, and we suspect it may be related to the creation of new tensor objects in Python and the related memory allocation on the backend.

Here are some stacktrace

#0  0x00007f7247c7f558 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#1  0x00007f7247ddc2cf in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#2  0x00007f7247ddc554 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#3  0x00007f7247ccb06f in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#4  0x00007f7247ccb1d9 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#5  0x00007f7247bc2bcd in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#6  0x00007f7247d88439 in cuStreamSynchronize () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#7  0x00007f71ef9a1c90 in ?? () from /usr/lib/python3.9/site-packages/torch/lib/libcudart-e409450e.so.11.0
#8  0x00007f71ef9f92e8 in cudaStreamSynchronize () from /usr/lib/python3.9/site-packages/torch/lib/libcudart-e409450e.so.11.0
#9  0x00007f71f17f8ccf in at::native::copy_kernel_cuda(at::TensorIterator&, bool) () from /usr/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so
#10 0x00007f722f1b7aa0 in at::native::copy_impl(at::Tensor&, at::Tensor const&, bool) () from /usr/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#11 0x00007f722f1b8c32 in at::native::copy_(at::Tensor&, at::Tensor const&, bool) () from /usr/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#12 0x00007f722fe27bdf in at::_ops::copy_::call(at::Tensor&, at::Tensor const&, bool) () from /usr/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#13 0x00007f722f4ab6cb in at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) () from /usr/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#14 0x00007f723018e48b in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>), &at::(anonymous namespace)::(anonymous namespace)::wrapper_CompositeExplicitAutograd___to_copy>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat> > >, at::Tensor (at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) ()
   from /usr/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#15 0x00007f722f975875 in at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) () from /usr/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#16 0x00007f722ffc7873 in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>), &at::(anonymous namespace)::_to_copy>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat> > >, at::Tensor (at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) () from /usr/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#17 0x00007f722f9fbc39 in at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) () from /usr/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#18 0x00007f722f4a2067 in at::native::to(at::Tensor const&, c10::Device, c10::ScalarType, bool, bool, c10::optional<c10::MemoryFormat>) () from /usr/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#19 0x00007f723035b3d9 in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, c10::Device, c10::ScalarType, bool, bool, c10::optional<c10::MemoryFormat>), &at::(anonymous namespace)::(anonymous namespace)::wrapper_CompositeImplicitAutograd_device_to>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, c10::Device, c10::ScalarType, bool, bool, c10::optional<c10::MemoryFormat> > >, at::Tensor (at::Tensor const&, c10::Device, c10::ScalarType, bool, bool, c10::optional<c10::MemoryFormat>)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, c10::Device, c10::ScalarType, bool, bool, c10::optional<c10::MemoryFormat>) () from /usr/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#20 0x00007f722fb7a92a in at::_ops::to_device::call(at::Tensor const&, c10::Device, c10::ScalarType, bool, bool, c10::optional<c10::MemoryFormat>) ()
   from /usr/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#21 0x00007f724741a52d in torch::utils::(anonymous namespace)::internal_new_from_data(c10::TensorOptions, c10::ScalarType, c10::optional<c10::Device>, _object*, bool, bool, bool, bool) ()
   from /usr/lib/python3.9/site-packages/torch/lib/libtorch_python.so
#22 0x00007f724741d8af in torch::utils::tensor_ctor(c10::DispatchKey, c10::ScalarType, torch::PythonArgs&) () from /usr/lib/python3.9/site-packages/torch/lib/libtorch_python.so
#23 0x00007f724708555e in torch::autograd::THPVariable_tensor(_object*, _object*, _object*) () from /usr/lib/python3.9/site-packages/torch/lib/libtorch_python.so
#24 0x00007f7268c8387d in cfunction_call (func=0x7f71ef95c950, args=<optimized out>, kwargs=<optimized out>) at Objects/methodobject.c:543
#25 0x00007f7268c56942 in _PyObject_MakeTpCall (tstate=0x555ce74040a0, callable=0x7f71ef95c950, args=<optimized out>, nargs=1, keywords=<optimized out>) at Objects/call.c:191
#26 0x00007f7268cd3e16 in _PyObject_VectorcallTstate (kwnames=0x7f71e9438730, nargsf=<optimized out>, args=0x555d48c17a18, callable=0x7f71ef95c950, tstate=<optimized out>) at ./Include/cpython/abstract.h:116
#27 _PyObject_VectorcallTstate (kwnames=0x7f71e9438730, nargsf=<optimized out>, args=0x555d48c17a18, callable=0x7f71ef95c950, tstate=<optimized out>) at ./Include/cpython/abstract.h:103
#28 PyObject_Vectorcall (kwnames=0x7f71e9438730, nargsf=<optimized out>, args=0x555d48c17a18, callable=0x7f71ef95c950) at ./Include/cpython/abstract.h:127
#29 call_function (kwnames=0x7f71e9438730, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at Python/ceval.c:5077
#30 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3537
#31 0x00007f7268cce2f4 in _PyEval_EvalFrame (throwflag=0, f=0x555d48c177f0, tstate=0x555ce74040a0) at ./Include/internal/pycore_ceval.h:40
#32 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=0x7f70069900a8, 
    kwargs=0x7f700a657f38, kwcount=<optimized out>, kwstep=1, defs=0x7f71e89fd8c8, defcount=5, kwdefs=0x0, closure=0x0, name=<optimized out>, qualname=0x7f71e8a163f0) at Python/ceval.c:4329
#33 0x00007f7268c570df in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:387
#34 0x00007f7268c58d27 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7f700a657f30, callable=0x7f71e84fbd30, tstate=0x555ce74040a0) at ./Include/cpython/abstract.h:118
--Type <RET> for more, q to quit, c to continue without paging--
#35 method_vectorcall (method=<optimized out>, args=0x7f700a657f38, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/classobject.c:53
#36 0x00007f7268c58049 in PyVectorcall_Call (callable=0x7f700a241e80, tuple=<optimized out>, kwargs=<optimized out>) at Objects/call.c:243
#37 0x00007f7268cd11aa in do_call_core (kwdict=0x7f700b7d60c0, callargs=0x7f7268133040, func=0x7f700a241e80, tstate=<optimized out>) at Python/ceval.c:5125
#38 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3582
#39 0x00007f7268cce2f4 in _PyEval_EvalFrame (throwflag=0, f=0x555d54dcb910, tstate=0x555ce74040a0) at ./Include/internal/pycore_ceval.h:40
#40 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=0x7f7006ecefa8, 
    kwargs=0x7f700ad3eb00, kwcount=<optimized out>, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=<optimized out>, qualname=0x7f71c7e10300) at Python/ceval.c:4329
#41 0x00007f7268c570df in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:387
#42 0x00007f7268c56665 in _PyObject_FastCallDictTstate (tstate=0x555ce74040a0, callable=0x7f71c7e25d30, args=<optimized out>, nargsf=<optimized out>, kwargs=<optimized out>) at Objects/call.c:129
#43 0x00007f7268c578ca in _PyObject_Call_Prepend (tstate=tstate@entry=0x555ce74040a0, callable=callable@entry=0x7f71c7e25d30, obj=obj@entry=0x7f7005c687f0, args=args@entry=0x7f7268133040, 
    kwargs=kwargs@entry=0x7f700bf89880) at Objects/call.c:489
#44 0x00007f7268c9d894 in slot_tp_call (self=0x7f7005c687f0, args=0x7f7268133040, kwds=0x7f700bf89880) at Objects/typeobject.c:6731
#45 0x00007f7268c56942 in _PyObject_MakeTpCall (tstate=0x555ce74040a0, callable=0x7f7005c687f0, args=<optimized out>, nargs=0, keywords=<optimized out>) at Objects/call.c:191
#46 0x00007f7268cd3e16 in _PyObject_VectorcallTstate (kwnames=0x7f71e8a165e0, nargsf=<optimized out>, args=0x555d2fafb980, callable=0x7f7005c687f0, tstate=<optimized out>) at ./Include/cpython/abstract.h:116
#47 _PyObject_VectorcallTstate (kwnames=0x7f71e8a165e0, nargsf=<optimized out>, args=0x555d2fafb980, callable=0x7f7005c687f0, tstate=<optimized out>) at ./Include/cpython/abstract.h:103
#48 PyObject_Vectorcall (kwnames=0x7f71e8a165e0, nargsf=<optimized out>, args=0x555d2fafb980, callable=0x7f7005c687f0) at ./Include/cpython/abstract.h:127
#49 call_function (kwnames=0x7f71e8a165e0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at Python/ceval.c:5077
#50 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3537
#51 0x00007f7268cce2f4 in _PyEval_EvalFrame (throwflag=0, f=0x555d2fafb7b0, tstate=0x555ce74040a0) at ./Include/internal/pycore_ceval.h:40
#52 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x7f700ad3e1a8, 
    kwcount=<optimized out>, kwstep=1, defs=0x7f71e8a16788, defcount=5, kwdefs=0x0, closure=0x0, name=<optimized out>, qualname=0x7f71e8a16710) at Python/ceval.c:4329
#53 0x00007f7268c57029 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:396
#54 0x00007f7268c58e4d in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=7, args=0x7f700ad3e170, callable=0x7f71e84fbca0, tstate=0x555ce74040a0) at ./Include/cpython/abstract.h:118
#55 method_vectorcall (method=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/classobject.c:83
#56 0x00007f7268c57ffc in PyVectorcall_Call (callable=0x7f700b7d63c0, tuple=<optimized out>, kwargs=<optimized out>) at Objects/call.c:231
#57 0x00007f7268cd11aa in do_call_core (kwdict=0x7f700bf73e00, callargs=0x7f7004507160, func=0x7f700b7d63c0, tstate=<optimized out>) at Python/ceval.c:5125
#58 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3582
#59 0x00007f7268cce2f4 in _PyEval_EvalFrame (throwflag=0, f=0x555d9ff9f3f0, tstate=0x555ce74040a0) at ./Include/internal/pycore_ceval.h:40
#60 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=7, kwnames=0x0, kwargs=0x7f7007f71d28, 
    kwcount=<optimized out>, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=<optimized out>, qualname=0x7f71c7e10300) at Python/ceval.c:4329
#61 0x00007f7268c5660b in _PyFunction_Vectorcall (kwnames=0x0, nargsf=<optimized out>, stack=<optimized out>, func=0x7f71c7e25d30) at Objects/call.c:396
#62 _PyObject_FastCallDictTstate (tstate=0x555ce74040a0, callable=0x7f71c7e25d30, args=<optimized out>, nargsf=<optimized out>, kwargs=<optimized out>) at Objects/call.c:118
#63 0x00007f7268c57a7e in _PyObject_Call_Prepend (tstate=tstate@entry=0x555ce74040a0, callable=callable@entry=0x7f71c7e25d30, obj=obj@entry=0x7f7005c70940, args=args@entry=0x7f7004507400, 
    kwargs=kwargs@entry=0x0) at Objects/call.c:489
#64 0x00007f7268c9d894 in slot_tp_call (self=self@entry=0x7f7005c70940, args=args@entry=0x7f7004507400, kwds=0x0) at Objects/typeobject.c:6731
#65 0x00007f7268c57f51 in _PyObject_Call (tstate=0x555ce74040a0, callable=0x7f7005c70940, args=0x7f7004507400, kwargs=<optimized out>) at Objects/call.c:281
#66 0x00007f7268cd11aa in do_call_core (kwdict=0x0, callargs=0x7f7004507400, func=0x7f7005c70940, tstate=<optimized out>) at Python/ceval.c:5125
#67 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3582
#68 0x00007f7268cce2f4 in _PyEval_EvalFrame (throwflag=0, f=0x7f7003e12dc0, tstate=0x555ce74040a0) at ./Include/internal/pycore_ceval.h:40
#69 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x7f70070670c8, 
    kwcount=<optimized out>, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x7f700a621780, name=<optimized out>, qualname=0x7f71e8a6adb0) at Python/ceval.c:4329
#70 0x00007f7268c57029 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:396
#71 0x00007f7268c57ffc in PyVectorcall_Call (callable=0x7f7003d501f0, tuple=<optimized out>, kwargs=<optimized out>) at Objects/call.c:231
#72 0x00007f7268cd11aa in do_call_core (kwdict=0x0, callargs=0x7f7007067090, func=0x7f7003d501f0, tstate=<optimized out>) at Python/ceval.c:5125
#73 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3582
#74 0x00007f7268cce2f4 in _PyEval_EvalFrame (throwflag=0, f=0x7f7003e0d640, tstate=0x555ce74040a0) at ./Include/internal/pycore_ceval.h:40
#75 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x7f712c48c330, 
    kwcount=<optimized out>, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=<optimized out>, qualname=0x7f71e8a224e0) at Python/ceval.c:4329
#76 0x00007f7268c57029 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:396
#77 0x00007f7268c57ffc in PyVectorcall_Call (callable=0x7f71e8a218b0, tuple=<optimized out>, kwargs=<optimized out>) at Objects/call.c:231
#78 0x00007f724707c1ca in THPFunction_apply(_object*, _object*) () from /usr/lib/python3.9/site-packages/torch/lib/libtorch_python.so
#79 0x00007f7268c838a0 in cfunction_call (func=func@entry=0x7f7006c2f950, args=args@entry=0x7f71500529a0, kwargs=<optimized out>) at Objects/methodobject.c:552
#80 0x00007f7268c57f51 in _PyObject_Call (tstate=0x555ce74040a0, callable=0x7f7006c2f950, args=0x7f71500529a0, kwargs=<optimized out>) at Objects/call.c:281
--Type <RET> for more, q to quit, c to continue without paging--
#81 0x00007f7268cd4642 in do_call_core (kwdict=0x7f70083921c0, callargs=0x7f71500529a0, func=0x7f7006c2f950, tstate=<optimized out>) at Python/ceval.c:5097
#82 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3582
#83 0x00007f7268cce2f4 in _PyEval_EvalFrame (throwflag=0, f=0x555d4ed6c820, tstate=0x555ce74040a0) at ./Include/internal/pycore_ceval.h:40
#84 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x7f700b0d4e28, 
    kwcount=<optimized out>, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x7f71c7d6d370, name=<optimized out>, qualname=0x7f71c7d63e70) at Python/ceval.c:4329
#85 0x00007f7268c57029 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:396
#86 0x00007f7268c58e4d in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=7, args=0x7f700b0d4df0, callable=0x7f71c7d2e820, tstate=0x555ce74040a0) at ./Include/cpython/abstract.h:118
#87 method_vectorcall (method=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/classobject.c:83
#88 0x00007f7268c57ffc in PyVectorcall_Call (callable=0x7f7008661880, tuple=<optimized out>, kwargs=<optimized out>) at Objects/call.c:231
#89 0x00007f7268cd11aa in do_call_core (kwdict=0x0, callargs=0x7f7003d2a6a0, func=0x7f7008661880, tstate=<optimized out>) at Python/ceval.c:5125
#90 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3582
#91 0x00007f7268cce2f4 in _PyEval_EvalFrame (throwflag=0, f=0x555d4f2191b0, tstate=0x555ce74040a0) at ./Include/internal/pycore_ceval.h:40
#92 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x555d4f912b80, 
    kwcount=<optimized out>, kwstep=1, defs=0x0, defcount=0, kwdefs=0x7f71e8a237c0, closure=0x0, name=<optimized out>, qualname=0x7f71c512a6f0) at Python/ceval.c:4329
#93 0x00007f7268c570df in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:387
#94 0x00007f7268cd3600 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775813, args=<optimized out>, callable=0x7f71e8a21820, tstate=0x555ce74040a0) at ./Include/cpython/abstract.h:118
#95 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775813, args=<optimized out>, callable=<optimized out>) at ./Include/cpython/abstract.h:127
#96 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x555ce74040a0) at Python/ceval.c:5077
#97 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3489
#98 0x00007f7268cce2f4 in _PyEval_EvalFrame (throwflag=0, f=0x555d4f9128e0, tstate=0x555ce74040a0) at ./Include/internal/pycore_ceval.h:40
#99 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=0x7f7003d37218, 
    kwargs=0x7f7006ae3628, kwcount=<optimized out>, kwstep=1, defs=0x7f71e8a61ad8, defcount=9, kwdefs=0x0, closure=0x0, name=<optimized out>, qualname=0x7f71e8a1b1c0) at Python/ceval.c:4329
#100 0x00007f7268c570df in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:387
#101 0x00007f7268c58d27 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7f7006ae3620, callable=0x7f71e84fe160, tstate=0x555ce74040a0) at ./Include/cpython/abstract.h:118
#102 method_vectorcall (method=<optimized out>, args=0x7f7006ae3628, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/classobject.c:53
#103 0x00007f7268c58049 in PyVectorcall_Call (callable=0x7f700a394580, tuple=<optimized out>, kwargs=<optimized out>) at Objects/call.c:243
#104 0x00007f7268cd11aa in do_call_core (kwdict=0x7f700b372f40, callargs=0x7f7268133040, func=0x7f700a394580, tstate=<optimized out>) at Python/ceval.c:5125
#105 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3582
#106 0x00007f7268cce2f4 in _PyEval_EvalFrame (throwflag=0, f=0x555d5506ea00, tstate=0x555ce74040a0) at ./Include/internal/pycore_ceval.h:40
#107 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=0x7f70039d0ad8, 
    kwargs=0x7f700447f1c0, kwcount=<optimized out>, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=<optimized out>, qualname=0x7f71c7e10300) at Python/ceval.c:4329
#108 0x00007f7268c570df in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:387
#109 0x00007f7268c56665 in _PyObject_FastCallDictTstate (tstate=0x555ce74040a0, callable=0x7f71c7e25d30, args=<optimized out>, nargsf=<optimized out>, kwargs=<optimized out>) at Objects/call.c:129
#110 0x00007f7268c578ca in _PyObject_Call_Prepend (tstate=tstate@entry=0x555ce74040a0, callable=callable@entry=0x7f71c7e25d30, obj=obj@entry=0x7f7003485a90, args=args@entry=0x7f7268133040, 
    kwargs=kwargs@entry=0x7f700bd65700) at Objects/call.c:489
#111 0x00007f7268c9d894 in slot_tp_call (self=0x7f7003485a90, args=0x7f7268133040, kwds=0x7f700bd65700) at Objects/typeobject.c:6731
#112 0x00007f7268c56942 in _PyObject_MakeTpCall (tstate=0x555ce74040a0, callable=0x7f7003485a90, args=<optimized out>, nargs=0, keywords=<optimized out>) at Objects/call.c:191
#113 0x00007f7268cd3e16 in _PyObject_VectorcallTstate (kwnames=0x7f71e8a61dd0, nargsf=<optimized out>, args=0x555d426c0b70, callable=0x7f7003485a90, tstate=<optimized out>) at ./Include/cpython/abstract.h:116
#114 _PyObject_VectorcallTstate (kwnames=0x7f71e8a61dd0, nargsf=<optimized out>, args=0x555d426c0b70, callable=0x7f7003485a90, tstate=<optimized out>) at ./Include/cpython/abstract.h:103
#115 PyObject_Vectorcall (kwnames=0x7f71e8a61dd0, nargsf=<optimized out>, args=0x555d426c0b70, callable=0x7f7003485a90) at ./Include/cpython/abstract.h:127
#116 call_function (kwnames=0x7f71e8a61dd0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at Python/ceval.c:5077
#117 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3537
#118 0x00007f7268cce2f4 in _PyEval_EvalFrame (throwflag=0, f=0x555d426c0970, tstate=0x555ce74040a0) at ./Include/internal/pycore_ceval.h:40
#119 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=0x7f700bda6e18, 
    kwargs=0x7f70041c05f8, kwcount=<optimized out>, kwstep=1, defs=0x7f71e8a1e158, defcount=10, kwdefs=0x0, closure=0x0, name=<optimized out>, qualname=0x7f71e8a1b760) at Python/ceval.c:4329
#120 0x00007f7268c570df in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:387
#121 0x00007f7268c58d27 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7f70041c05f0, callable=0x7f71e84fe940, tstate=0x555ce74040a0) at ./Include/cpython/abstract.h:118
#122 method_vectorcall (method=<optimized out>, args=0x7f70041c05f8, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/classobject.c:53
#123 0x00007f7268c58049 in PyVectorcall_Call (callable=0x7f7061b71c80, tuple=<optimized out>, kwargs=<optimized out>) at Objects/call.c:243
#124 0x00007f7268cd11aa in do_call_core (kwdict=0x7f700a762280, callargs=0x7f7268133040, func=0x7f7061b71c80, tstate=<optimized out>) at Python/ceval.c:5125
#125 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3582
#126 0x00007f7268cce2f4 in _PyEval_EvalFrame (throwflag=0, f=0x555d5506e4f0, tstate=0x555ce74040a0) at ./Include/internal/pycore_ceval.h:40
#127 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=0x7f700b8aecd8, 
--Type <RET> for more, q to quit, c to continue without paging--
    kwargs=0x7f7009ea2850, kwcount=<optimized out>, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=<optimized out>, qualname=0x7f71c7e10300) at Python/ceval.c:4329
#128 0x00007f7268c570df in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:387
#129 0x00007f7268c56665 in _PyObject_FastCallDictTstate (tstate=0x555ce74040a0, callable=0x7f71c7e25d30, args=<optimized out>, nargsf=<optimized out>, kwargs=<optimized out>) at Objects/call.c:129
#130 0x00007f7268c578ca in _PyObject_Call_Prepend (tstate=tstate@entry=0x555ce74040a0, callable=callable@entry=0x7f71c7e25d30, obj=obj@entry=0x7f70034c68b0, args=args@entry=0x7f7268133040, 
    kwargs=kwargs@entry=0x7f700bf0aa80) at Objects/call.c:489
#131 0x00007f7268c9d894 in slot_tp_call (self=self@entry=0x7f70034c68b0, args=args@entry=0x7f7268133040, kwds=0x7f700bf0aa80) at Objects/typeobject.c:6731
#132 0x00007f7268c57f51 in _PyObject_Call (tstate=0x555ce74040a0, callable=0x7f70034c68b0, args=0x7f7268133040, kwargs=<optimized out>) at Objects/call.c:281
#133 0x00007f7268cd11aa in do_call_core (kwdict=0x7f700bf0aa80, callargs=0x7f7268133040, func=0x7f70034c68b0, tstate=<optimized out>) at Python/ceval.c:5125
#134 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3582
#135 0x00007f7268cce2f4 in _PyEval_EvalFrame (throwflag=0, f=0x7f7003e0d840, tstate=0x555ce74040a0) at ./Include/internal/pycore_ceval.h:40
#136 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=0x7f70088b16d8, 
    kwargs=0x7f7009ea2670, kwcount=<optimized out>, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=<optimized out>, qualname=0x7f71e8ac8260) at Python/ceval.c:4329
#137 0x00007f7268c570df in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:387
#138 0x00007f7268c58049 in PyVectorcall_Call (callable=0x7f71e8a59af0, tuple=<optimized out>, kwargs=<optimized out>) at Objects/call.c:243
#139 0x00007f7268cd11aa in do_call_core (kwdict=0x7f700878c080, callargs=0x7f700aace550, func=0x7f71e8a59af0, tstate=<optimized out>) at Python/ceval.c:5125
#140 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3582
#141 0x00007f7268cce2f4 in _PyEval_EvalFrame (throwflag=0, f=0x555d4f158f80, tstate=0x555ce74040a0) at ./Include/internal/pycore_ceval.h:40
#142 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=0x7f700a910a98, 
    kwargs=0x7f70041c0c38, kwcount=<optimized out>, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x7f71e8ab1af0, name=<optimized out>, qualname=0x7f71e95eb690) at Python/ceval.c:4329
#143 0x00007f7268c570df in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:387
#144 0x00007f7268c58d27 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7f70041c0c30, callable=0x7f71e8a59b80, tstate=0x555ce74040a0) at ./Include/cpython/abstract.h:118
#145 method_vectorcall (method=<optimized out>, args=0x7f70041c0c38, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/classobject.c:53
#146 0x00007f7268c58049 in PyVectorcall_Call (callable=0x7f7008661700, tuple=<optimized out>, kwargs=<optimized out>) at Objects/call.c:243
#147 0x00007f7268cd11aa in do_call_core (kwdict=0x7f7008a51280, callargs=0x7f7268133040, func=0x7f7008661700, tstate=<optimized out>) at Python/ceval.c:5125
#148 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3582
#149 0x00007f7268cce2f4 in _PyEval_EvalFrame (throwflag=0, f=0x555d4b3934d0, tstate=0x555ce74040a0) at ./Include/internal/pycore_ceval.h:40
#150 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=0x7f700a241318, 
    kwargs=0x7f7009ea22b0, kwcount=<optimized out>, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=<optimized out>, qualname=0x7f71c7e10300) at Python/ceval.c:4329
#151 0x00007f7268c570df in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:387
#152 0x00007f7268c56665 in _PyObject_FastCallDictTstate (tstate=0x555ce74040a0, callable=0x7f71c7e25d30, args=<optimized out>, nargsf=<optimized out>, kwargs=<optimized out>) at Objects/call.c:129
#153 0x00007f7268c578ca in _PyObject_Call_Prepend (tstate=tstate@entry=0x555ce74040a0, callable=callable@entry=0x7f71c7e25d30, obj=obj@entry=0x7f7150030850, args=args@entry=0x7f7268133040, 
    kwargs=kwargs@entry=0x7f7008661f00) at Objects/call.c:489
#154 0x00007f7268c9d894 in slot_tp_call (self=self@entry=0x7f7150030850, args=args@entry=0x7f7268133040, kwds=0x7f7008661f00) at Objects/typeobject.c:6731
#155 0x00007f7268c57f51 in _PyObject_Call (tstate=0x555ce74040a0, callable=0x7f7150030850, args=0x7f7268133040, kwargs=<optimized out>) at Objects/call.c:281
#156 0x00007f7268cd11aa in do_call_core (kwdict=0x7f7008661f00, callargs=0x7f7268133040, func=0x7f7150030850, tstate=<optimized out>) at Python/ceval.c:5125
#157 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3582
#158 0x00007f7268c57273 in _PyEval_EvalFrame (throwflag=0, f=0x555d54dba640, tstate=0x555ce74040a0) at ./Include/internal/pycore_ceval.h:40
#159 function_code_fastcall (tstate=0x555ce74040a0, co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>) at Objects/call.c:330
#160 0x00007f7268ccf705 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x555cf0050298, callable=0x7f71e84b13a0, tstate=0x555ce74040a0) at ./Include/cpython/abstract.h:118
#161 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x555cf0050298, callable=<optimized out>) at ./Include/cpython/abstract.h:127
#162 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x555ce74040a0) at Python/ceval.c:5077
#163 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3506
#164 0x00007f7268c57273 in _PyEval_EvalFrame (throwflag=0, f=0x555cf004fff0, tstate=0x555ce74040a0) at ./Include/internal/pycore_ceval.h:40
#165 function_code_fastcall (tstate=0x555ce74040a0, co=<optimized out>, args=<optimized out>, nargs=0, globals=<optimized out>) at Objects/call.c:330
#166 0x00007f7268ccf454 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x555ce74656c0, callable=0x7f71904b8e50, tstate=0x555ce74040a0) at ./Include/cpython/abstract.h:118
#167 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x555ce74656c0, callable=<optimized out>) at ./Include/cpython/abstract.h:127
#168 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x555ce74040a0) at Python/ceval.c:5077
#169 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3520
#170 0x00007f7268ccd8a1 in _PyEval_EvalFrame (throwflag=0, f=0x555ce7465550, tstate=0x555ce74040a0) at ./Include/internal/pycore_ceval.h:40
#171 _PyEval_EvalCode (qualname=0x0, name=<optimized out>, closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwstep=2, kwcount=<optimized out>, kwargs=0x0, kwnames=<optimized out>, argcount=<optimized out>, 
    args=<optimized out>, locals=<optimized out>, globals=<optimized out>, _co=<optimized out>, tstate=0x555ce74040a0) at Python/ceval.c:4329
#172 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=0x0, kwcount=0, kwstep=2, 
    defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:4361
--Type <RET> for more, q to quit, c to continue without paging--
#173 0x00007f7268ccd5f3 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:4377
#174 0x00007f7268ccd5ab in PyEval_EvalCode (co=co@entry=0x7f7267f90a80, globals=globals@entry=0x7f72680aa640, locals=locals@entry=0x7f72680aa640) at Python/ceval.c:828
#175 0x00007f7268d5fecd in run_eval_code_obj (tstate=0x555ce74040a0, co=0x7f7267f90a80, globals=0x7f72680aa640, locals=0x7f72680aa640) at Python/pythonrun.c:1221
#176 0x00007f7268d5fe5a in run_mod (mod=<optimized out>, filename=<optimized out>, globals=0x7f72680aa640, locals=0x7f72680aa640, flags=<optimized out>, arena=<optimized out>) at Python/pythonrun.c:1242
#177 0x00007f7268bf4edc in pyrun_file (fp=fp@entry=0x555ce746db80, filename=filename@entry=0x7f7267fd8450, start=start@entry=257, globals=globals@entry=0x7f72680aa640, locals=locals@entry=0x7f72680aa640, 
    closeit=closeit@entry=1, flags=0x7fff37378d28) at Python/pythonrun.c:1140
#178 0x00007f7268bf4c6f in pyrun_simple_file (flags=0x7fff37378d28, closeit=1, filename=0x7f7267fd8450, fp=0x555ce746db80) at Python/pythonrun.c:450
#179 PyRun_SimpleFileExFlags (fp=fp@entry=0x555ce746db80, filename=<optimized out>, closeit=closeit@entry=1, flags=flags@entry=0x7fff37378d28) at Python/pythonrun.c:483
#180 0x00007f7268bf5b21 in PyRun_AnyFileExFlags (fp=fp@entry=0x555ce746db80, filename=<optimized out>, closeit=closeit@entry=1, flags=flags@entry=0x7fff37378d28) at Python/pythonrun.c:92
#181 0x00007f7268c00ddf in pymain_run_file (cf=0x7fff37378d28, config=0x555ce7400a00) at Modules/main.c:373
#182 pymain_run_python (exitcode=0x7fff37378d20) at Modules/main.c:598
#183 Py_RunMain () at Modules/main.c:677
#184 0x00007f7268d66039 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:731
#185 0x00007f726877a09b in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#186 0x0000555ce5f9f08a in _start ()

We would appreciate any insights or suggestions on how to resolve this issue. Thank you in advance for your help.

Start by reproducing Step 3 of the DeepSpeed Chat RLHF example with the listed Python, PyTorch, Transformers, GPU, and model versions. Inspect the forward-pass stack trace around PyTorch's THPVariable_tensor entry point and determine why execution remains in CUDA synchronization; done means Step 3 completes and proceeds to later steps.