[BUG] tensor values change across different stages of pipeline parallel

I'm using pipeline parallel for 4 different sub models.
model 1 and 2 are on stage=0, and the others are on stage=1 like this

stage=0 layers=2
     0: MyResnet
     1: MyEncoder_1
stage=1 layers=2
     2: DummyModel
     3: MyDecoder_1

in MyEncoder_1 on stage=0, everything is ok, below will be passed to the next module by the forward() of MyEncoder_1:

ic| max(token[0]): tensor(32651, device='cuda:0'), '@encoder'
ic| min(token[0]): tensor(0, device='cuda:0'), '@encoder'

but, DummyModel on the stage=1, received different value and type of input tensor:

ic| max(token[0]): tensor(64608., device='cuda:1', dtype=torch.float16, grad_fn=<UnbindBackward>)
    "@dummy": '@dummy'
ic| min(token[0]): tensor(0., device='cuda:1', dtype=torch.float16, grad_fn=<UnbindBackward>)
    "@dummy": '@dummy'

what can be the cause of this?
I'm using different devices: cuda:0=RTX2080Ti, and cuda:1=GTX1070.
thank you.

notice:
I made one change to deepspeed.runtime.pipe.engine.py locally, around line 681 , as I couldn't figure out how to pass multiple input from dataset to pipeline properly.

https://github.com/microsoft/DeepSpeed/blob/86b948f9ebbe63a600b2dcf04058c9fcc05efe9c/deepspeed/runtime/pipe/engine.py#L681

        batch = self._next_batch()
        if isinstance(batch[0], list):
            batch[0] = tuple(batch[0])

and my modules are written like this

class DummyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.dummy = torch.nn.Parameter(torch.zeros(1))

    def forward(self, x):
        _, token = x

        # ic(max(token[0]), "@dummy")
        # ic(min(token[0]), "@dummy")
        # ic(max(token[1]), "@dummy")
        # ic(min(token[1]), "@dummy")

        return _, token

ds_report output

--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
sparse_attn ............ [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
async_io ............... [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/atmin2/.pyenv/versions/3.8.5/lib/python3.8/site-packages/torch']
torch version .................... 1.7.0+cu110
torch cuda version ............... 11.0
nvcc version ..................... 11.0
deepspeed install path ........... ['/home/atmin2/crawler/seq_cvt/DeepSpeed/deepspeed']
deepspeed info ................... 0.5.2+74f058b, 74f058b, master
deepspeed wheel compiled w. ...... torch 1.7, cuda 11.0

Start by inspecting deepspeed/runtime/pipe/engine.py around line 681, including the local batch-handling change, and trace how the tuple from _next_batch() is transferred between pipeline stages. Reproduce the shown two-stage setup with the reported PyTorch and DeepSpeed versions; done means identifying why the receiving stage changes the tensor values or dtype and demonstrating preserved inputs across the stage boundary.