[BUG]requests_grad=False would be change to "True" when tensor pass from the first GPU to the second.

Describe the bug
I used only pipeline without any other parallel model. In the piped model, I packed sample and label as a tuple. The sample was calculated along each layer and label was passed to last layer to calculate loss. And I found a phenomenon that in the first device, label's requests_grad is always False and that's correct value. But when it has been passed to the next device, it's requests_grad is changed as True immediately. It thus then throw a error at backward as "RuntimeError: element 2 of tensors does not require grad and does not have a grad_fn" on first device.

My output

[2025-06-15 21:04:44][rank 0/8] torch.Size([2, 40962, 192]) True
[2025-06-15 21:04:44][rank 0/8] torch.Size([2, 40962, 192]) True
[2025-06-15 21:04:44][rank 0/8] last  torch.Size([2, 40962, 192]) True
[2025-06-15 21:04:44]**[rank 0/8]** **last  torch.Size([2, 1, 46, 720, 1440]) False**
[2025-06-15 21:04:44][rank 1/8] torch.Size([2, 40962, 192]) True
[2025-06-15 21:04:44][rank 1/8] torch.Size([2, 40962, 192]) True
[2025-06-15 21:04:44]**[rank 1/8]** **torch.Size([2, 1, 46, 720, 1440]) True**
[2025-06-15 21:04:44][rank 1/8] torch.Size([40962, 192]) True
[2025-06-15 21:04:44][rank 1/8] torch.Size([40962, 192]) True

Both bold are same tensor.

Error output

[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/extDisk/ZY/dmc/train.py", line 279, in <module>
[rank0]:     main(args)
[rank0]:   File "/home/extDisk/ZY/dmc/train.py", line 219, in main
[rank0]:     train_loss += model_engine.train_batch(data_iter=train_iter)
[rank0]:   File "/root/miniconda3/envs/DMC/lib/python3.10/site-packages/deepspeed/runtime/pipe/engine.py", line 389, in train_batch
[rank0]:     self._exec_schedule(sched)
[rank0]:   File "/root/miniconda3/envs/DMC/lib/python3.10/site-packages/deepspeed/runtime/pipe/engine.py", line 1422, in _exec_schedule
[rank0]:     self._exec_instr(**cmd.kwargs)
[rank0]:   File "/root/miniconda3/envs/DMC/lib/python3.10/site-packages/deepspeed/runtime/pipe/engine.py", line 860, in _exec_backward_pass
[rank0]:     torch.autograd.backward(tensors=out_tensors, grad_tensors=grad_tensors)
[rank0]:   File "/root/miniconda3/envs/DMC/lib/python3.10/site-packages/torch/autograd/__init__.py", line 267, in backward
[rank0]:     _engine_run_backward(
[rank0]:   File "/root/miniconda3/envs/DMC/lib/python3.10/site-packages/torch/autograd/graph.py", line 744, in _engine_run_backward
[rank0]:     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank0]: RuntimeError: element 2 of tensors does not require grad and does not have a grad_fn
/root/miniconda3/envs/DMC/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 4 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

Is that a bug that set requests_grad=True arbitrarily after recv tensor from previous device?

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

ds_report output
Please run ds_report to give us details about your setup.
[2025-06-15 21:37:54,183] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)

DeepSpeed C++/CUDA extension op report

NOTE: Ops not installed will be just-in-time (JIT) compiled at
runtime if needed. Op compatibility means that your system
meet the required dependencies to JIT install the op.

JIT compiled ops requires ninja
ninja .................. [OKAY]

op name ................ installed .. compatible

[WARNING] async_io requires the dev libaio .so object and headers but these were not found.
[WARNING] async_io: please install the libaio-devel package with yum
[WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
fused_adam ............. [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_lion ............... [NO] ....... [OKAY]
[WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
evoformer_attn ......... [NO] ....... [NO]
fp_quantizer ........... [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
fused_lion ............. [NO] ....... [OKAY]
/root/miniconda3/envs/DMC/compiler_compat/ld: warning: libstdc++.so.6, needed by /usr/local/cuda-12.1/lib64/libcufile.so, not found (try using -rpath or -rpath-link)
/root/miniconda3/envs/DMC/compiler_compat/ld: warning: libm.so.6, needed by /usr/local/cuda-12.1/lib64/libcufile.so, not found (try using -rpath or -rpath-link)
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::runtime_error::~runtime_error()@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to __gxx_personality_v0@CXXABI_1.3'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::ostream::tellp()@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::chrono::_V2::steady_clock::now()@GLIBCXX_3.4.19'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::string::_M_replace_aux(unsigned long, unsigned long, unsigned long, char)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to dlopen'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to typeinfo for bool@CXXABI_1.3' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::__throw_logic_error(char const*)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to VTT for std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::logic_error@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::locale::~locale()@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_string<char, std::char_traits, std::allocator >::basic_string(std::string const&, unsigned long, unsigned long)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to __cxa_end_catch@CXXABI_1.3' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to VTT for std::basic_ofstream<char, std::char_traits >@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::logic_error::~logic_error()@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for __cxxabiv1::__si_class_type_info@CXXABI_1.3'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_ios<char, std::char_traits<char> >::_M_cache_locale(std::locale const&)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to VTT for std::basic_stringstream<char, std::char_traits, std::allocator >@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to operator new[](unsigned long)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::string::_M_leak_hard()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::basic_ifstream<char, std::char_traits<char> >@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_streambuf<wchar_t, std::char_traits<wchar_t> >::basic_streambuf(std::basic_streambuf<wchar_t, std::char_traits<wchar_t> > const&)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::string::append(char const*, unsigned long)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_string<char, std::char_traits, std::allocator >::basic_string(std::string const&)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to typeinfo for unsigned short@CXXABI_1.3' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::string::resize(unsigned long, char)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to typeinfo for char const*@CXXABI_1.3' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::ctype::_M_widen_init() const@GLIBCXX_3.4.11'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::__throw_invalid_argument(char const*)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::locale::operator=(std::locale const&)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_ios<wchar_t, std::char_traits<wchar_t> >::_M_cache_locale(std::locale const&)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::_Rb_tree_decrement(std::_Rb_tree_node_base const*)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to __cxa_free_exception@CXXABI_1.3' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::condition_variable::notify_one()@GLIBCXX_3.4.11'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::ios_base::Init::~Init()@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_string<char, std::char_traits, std::allocator >::~basic_string()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to __cxa_pure_virtual@CXXABI_1.3' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::ostream::flush()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for __cxxabiv1::__class_type_info@CXXABI_1.3' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to __cxa_rethrow@CXXABI_1.3'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_fstream<char, std::char_traits >::~basic_fstream()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::string::compare(char const*) const@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to VTT for std::basic_ostringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::locale::locale()@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::chrono::_V2::system_clock::now()@GLIBCXX_3.4.19'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to VTT for std::basic_ifstream<char, std::char_traits<char> >@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::_Hash_bytes(void const*, unsigned long, unsigned long)@CXXABI_1.3.5'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::ostream& std::ostream::_M_insert<long long>(long long)@GLIBCXX_3.4.9' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to typeinfo for char*@CXXABI_1.3'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::__detail::_Prime_rehash_policy::_M_need_rehash(unsigned long, unsigned long, unsigned long) const@GLIBCXX_3.4.18' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::out_of_range@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to dlclose' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::ostream& std::ostream::_M_insert(unsigned long)@GLIBCXX_3.4.9'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::_Rb_tree_increment(std::_Rb_tree_node_base const*)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::ios_base::ios_base()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::range_error::~range_error()@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::__basic_file::__basic_file()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to __cxa_guard_acquire@CXXABI_1.3' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::ostream& std::ostream::_M_insert(bool)@GLIBCXX_3.4.9'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::overflow_error@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to VTT for std::basic_fstream<char, std::char_traits >@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::range_error@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::basic_ios<char, std::char_traits >@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::basic_filebuf<char, std::char_traits<char> >@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to operator delete@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_string<char, std::char_traits, std::allocator >::basic_string(unsigned long, char, std::allocator const&)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::__detail::_List_node_base::_M_transfer(std::__detail::_List_node_base*, std::__detail::_List_node_base*)@GLIBCXX_3.4.15' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::string::replace(unsigned long, unsigned long, char const*, unsigned long)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to typeinfo for std::exception@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::_Rep::_M_destroy(std::allocator<wchar_t> const&)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::istream& std::istream::_M_extract<double>(double&)@GLIBCXX_3.4.9' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_filebuf<char, std::char_traits >::close()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::basic_fstream<char, std::char_traits<char> >@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_ifstream<char, std::char_traits >::basic_ifstream(char const*, std::_Ios_Openmode)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::string::append(std::string const&)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to operator new(unsigned long)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to VTT for std::basic_istringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to typeinfo for unsigned int@CXXABI_1.3'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::string::append(char const*)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::domain_error@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::string::find(char, unsigned long) const@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::ostream::put(char)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to typeinfo for int@CXXABI_1.3' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::__throw_bad_alloc()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to __cxa_thread_atexit@CXXABI_1.3.7' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::_Rb_tree_increment(std::_Rb_tree_node_base*)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_ifstream<char, std::char_traits<char> >::~basic_ifstream()@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::ios_base::Init::Init()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::condition_variable::condition_variable()@GLIBCXX_3.4.11' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_filebuf<char, std::char_traits >::basic_filebuf()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to VTT for std::basic_istringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::domain_error::~domain_error()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::cerr@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::string::find(char const*, unsigned long, unsigned long) const@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::basic_istringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_string<char, std::char_traits, std::allocator >::basic_string(std::allocator const&)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::str() const@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::invalid_argument@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to typeinfo for void*@CXXABI_1.3' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::string::assign(std::string const&)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_ostringstream()@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to typeinfo for unsigned long@CXXABI_1.3' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::__detail::_List_node_base::_M_hook(std::__detail::_List_node_base*)@GLIBCXX_3.4.15'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::__detail::_List_node_base::_M_unhook()@GLIBCXX_3.4.15' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::basic_ostringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::_M_sync(char*, unsigned long, unsigned long)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_iostream<char, std::char_traits >::~basic_iostream()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::locale::locale(std::locale const&)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::basic_istringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to log2f@GLIBC_2.2.5' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::ostream::operator<<(std::basic_streambuf<char, std::char_traits >)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::basic_streambuf<wchar_t, std::char_traits<wchar_t> >@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::exception::~exception()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::__basic_file::is_open() const@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to dlerror' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_istringstream<char, std::char_traits, std::allocator >::~basic_istringstream()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::string::swap(std::string&)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to typeinfo for unsigned long@CXXABI_1.3'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_streambuf<char, std::char_traits >::basic_streambuf(std::basic_streambuf<char, std::char_traits > const&)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_ios<char, std::char_traits<char> >::init(std::basic_streambuf<char, std::char_traits<char> >*)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to dlsym'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::__throw_bad_cast()@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_ios<char, std::char_traits >::clear(std::_Ios_Iostate)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_streambuf<wchar_t, std::char_traits<wchar_t> >::operator=(std::basic_streambuf<wchar_t, std::char_traits<wchar_t> > const&)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to operator delete(void*)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::ostream::operator<<(int)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::string::_Rep::_S_empty_rep_storage@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::string::_Rep::_M_destroy(std::allocator<char> const&)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_iostream<wchar_t, std::char_traits<wchar_t> >::~basic_iostream()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::runtime_error@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::basic_ofstream<char, std::char_traits >@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::_Rb_tree_insert_and_rebalance(bool, std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_stringstream<char, std::char_traits, std::allocator >::~basic_stringstream()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to VTT for std::basic_stringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::ostream& std::ostream::_M_insert(long)@GLIBCXX_3.4.9'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::istream::get()@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to typeinfo for unsigned long long@CXXABI_1.3'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::out_of_range::~out_of_range()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::length_error::~length_error()@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_ostream<char, std::char_traits >& std::__ostream_insert<char, std::char_traits >(std::basic_ostream<char, std::char_traits >&, char const*, long)@GLIBCXX_3.4.9'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::invalid_argument::~invalid_argument()@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::swap(std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >&)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::cout@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::ostream& std::ostream::_M_insert(unsigned long long)@GLIBCXX_3.4.9'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to typeinfo for int*@CXXABI_1.3' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::ostream& std::ostream::_M_insert<void const*>(void const*)@GLIBCXX_3.4.9'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::underflow_error@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::basic_streambuf<char, std::char_traits >@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to typeinfo for std::out_of_range@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to __cxa_allocate_exception@CXXABI_1.3'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::basic_ios<wchar_t, std::char_traits<wchar_t> >@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to typeinfo for void const*@CXXABI_1.3'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_ios<wchar_t, std::char_traits<wchar_t> >::init(std::basic_streambuf<wchar_t, std::char_traits<wchar_t> >*)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::string::reserve(unsigned long)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to __cxa_begin_catch@CXXABI_1.3' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to typeinfo for long@CXXABI_1.3'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::_Rep::_S_empty_rep_storage@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::string::_M_leak()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_filebuf<char, std::char_traits<char> >::open(char const*, std::_Ios_Openmode)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_stringbuf<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::_M_sync(wchar_t*, unsigned long, unsigned long)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::istream::getline(char*, long, char)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_istream<char, std::char_traits >& std::getline<char, std::char_traits, std::allocator >(std::basic_istream<char, std::char_traits >&, std::basic_string<char, std::char_traits, std::allocator >&, char)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::basic_stringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::condition_variable::~condition_variable()@GLIBCXX_3.4.11'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::basic_stringbuf<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::string::insert(unsigned long, char const*, unsigned long)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::string::assign(char const*, unsigned long)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to typeinfo for unsigned char@CXXABI_1.3'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::ios_base::ios_base()@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::__throw_out_of_range(char const*)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::overflow_error::~overflow_error()@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::__throw_length_error(char const*)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::__throw_system_error(int)@GLIBCXX_3.4.11' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_ofstream<char, std::char_traits >::close()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::ostream& std::ostream::_M_insert<double>(double)@GLIBCXX_3.4.9' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_streambuf<char, std::char_traits >::operator=(std::basic_streambuf<char, std::char_traits > const&)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to typeinfo for long long@CXXABI_1.3' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_string<char, std::char_traits, std::allocator >::basic_string(char const*, unsigned long, std::allocator const&)@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::basic_ifstream<char, std::char_traits<char> >::close()@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to __cxa_guard_release@CXXABI_1.3'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to __cxa_throw@CXXABI_1.3' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::underflow_error::~underflow_error()@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to std::_Rb_tree_decrement(std::_Rb_tree_node_base*)@GLIBCXX_3.4' /root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to vtable for std::length_error@GLIBCXX_3.4'
/root/miniconda3/envs/DMC/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to `std::basic_filebuf<char, std::char_traits >::~basic_filebuf()@GLIBCXX_3.4'
collect2: 错误：ld 返回 1
gds .................... [NO] ....... [NO]
transformer_inference .. [NO] ....... [OKAY]
inference_core_ops ..... [NO] ....... [OKAY]
cutlass_ops ............ [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
ragged_device_ops ...... [NO] ....... [OKAY]
ragged_ops ............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
[WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
[WARNING] using untested triton version (2.3.1), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]

DeepSpeed general environment info:
torch install path ............... ['/root/miniconda3/envs/DMC/lib/python3.10/site-packages/torch']
torch version .................... 2.3.1
deepspeed install path ........... ['/root/miniconda3/envs/DMC/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.15.4, unknown, unknown
torch cuda version ............... 12.1
torch hip version ................ None
nvcc version ..................... 12.1
deepspeed wheel compiled w. ...... torch 2.3, cuda 12.1
shared memory (/dev/shm) size .... 62.77 GB
(DMC) [root@localhost dmc]#

Screenshots
If applicable, add screenshots to help explain your problem.

System info (please complete the following information):

• OS: [e.g. Ubuntu 18.04]
• GPU count and types [e.g. two machines with x8 A100s each]
• Interconnects (if applicable) [e.g., two machines connected with 100 Gbps IB]
• Python version
• Any other relevant info about your setup

Launcher context
Are you launching your experiment with the deepspeed launcher, MPI, or something else?

Docker context
Are you using a specific docker image that you can share?

Additional context
Add any other context about the problem here.

Start with deepspeed/runtime/pipe/engine.py, especially _exec_backward_pass, and trace how tuple elements are transferred between pipeline stages. Reproduce the reported case with a sample and label tuple, then verify whether the label's requires_grad state changes after transfer and whether backward completes without the reported RuntimeError.