Deadlock on `log_dict` with different keys across ranks

Bug description

Logging dictionaries across ranks with different keys lead to NCCL silently dying. The expected behavior is for only the existing keys across dictionaries to be averaged.

What version are you seeing the problem on?

v2.0

How to reproduce the bug

Use multiple GPUs!

import pytorch_lightning as pl
import torch
import random

class SimpleNetwork(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(32, 2)

    def forward(self, x):
        return self.layer(x)

    def training_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self(x)
        loss = torch.nn.functional.mse_loss(y_hat, y)
        return loss

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.02)

    def train_dataloader(self):
        return torch.utils.data.DataLoader(
            torch.utils.data.TensorDataset(torch.randn(100, 32), torch.randn(100, 2)),
            batch_size=32,
        )
    
    # use log_dict to log metrics
    def training_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self(x)
        loss = torch.nn.functional.mse_loss(y_hat, y)
        dict_to_log = {'loss': loss}
        randomly_add_another_metric = random.random()
        if randomly_add_another_metric > 0.5:
            dict_to_log['another_metric'] = torch.tensor(random.random())
        self.log_dict(
            dict_to_log,
            on_step=False,
            on_epoch=True,
            sync_dist=True,
            batch_size=32,
        )
        return loss

def train_network():
    # Initialize the Lightning Trainer
    trainer = pl.Trainer(
        max_epochs=10,
    )

    # Create an instance of your SimpleNetwork
    model = SimpleNetwork()

    # Start the training
    trainer.fit(model)

train_network()

Error messages and logs

[I debug.cpp:49] [c10d] The debug level is set to DETAIL.
/myenv/python3.10/site-packages/lightning_fabric/plugins/environments/slurm.py:191: The `srun` command is available on your system but is not used. HINT: If your intention is to run Lightning on SLURM, prepend your python command with `srun` like so: srun python examples/nccl_bug.py ...
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
/myenv/python3.10/site-packages/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py:67: Starting from v1.9.0, `tensorboardX` has been removed as a dependency of the `pytorch_lightning` package, due to potential conflicts with other packages in the ML ecosystem. For this reason, `logger=True` will use `CSVLogger` as the default logger, unless the `tensorboard` or `tensorboardX` packages are found. Please `pip install lightning[extra]` or one of them to enable TensorBoard support by default
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/4
[I socket.cpp:452] [c10d - debug] The server socket will attempt to listen on an IPv6 address.
[I socket.cpp:502] [c10d - debug] The server socket is attempting to listen on [::]:32903.
[I socket.cpp:576] [c10d] The server socket has started to listen on [::]:32903.
[I TCPStore.cpp:252] [c10d - debug] The server has started on port = 32903.
[I socket.cpp:686] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (127.0.0.1, 32903).
[I socket.cpp:761] [c10d - trace] The client socket is attempting to connect to [localhost]:32903.
[I socket.cpp:849] [c10d] The client socket has connected to [localhost]:32903 on [localhost]:49152.
[I TCPStore.cpp:261] [c10d - debug] TCP client connected to host 127.0.0.1:32903
[I socket.cpp:297] [c10d - debug] The server socket on [::]:32903 has accepted a connection from [localhost]:49152.
[I debug.cpp:49] [c10d] The debug level is set to DETAIL.
[I debug.cpp:49] [c10d] The debug level is set to DETAIL.
[I debug.cpp:49] [c10d] The debug level is set to DETAIL.
Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/4
[I socket.cpp:686] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (127.0.0.1, 32903).
[I socket.cpp:761] [c10d - trace] The client socket is attempting to connect to [localhost]:32903.
[I socket.cpp:297] [c10d - debug] The server socket on [::]:32903 has accepted a connection from [localhost]:49156.
[I socket.cpp:849] [c10d] The client socket has connected to [localhost]:32903 on [localhost]:49156.
[I TCPStore.cpp:261] [c10d - debug] TCP client connected to host 127.0.0.1:32903
Initializing distributed: GLOBAL_RANK: 3, MEMBER: 4/4
[I socket.cpp:686] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (127.0.0.1, 32903).
[I socket.cpp:761] [c10d - trace] The client socket is attempting to connect to [localhost]:32903.
[I socket.cpp:849] [c10d] The client socket has connected to [localhost]:32903 on [localhost]:49168.
[I TCPStore.cpp:261] [c10d - debug] TCP client connected to host 127.0.0.1:32903
[I socket.cpp:297] [c10d - debug] The server socket on [::]:32903 has accepted a connection from [localhost]:49168.
Initializing distributed: GLOBAL_RANK: 2, MEMBER: 3/4
[I socket.cpp:686] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (127.0.0.1, 32903).
[I socket.cpp:761] [c10d - trace] The client socket is attempting to connect to [localhost]:32903.
[I socket.cpp:849] [c10d] The client socket has connected to [localhost]:32903 on [localhost]:49180.
[I TCPStore.cpp:261] [c10d - debug] TCP client connected to host 127.0.0.1:32903
[I socket.cpp:297] [c10d - debug] The server socket on [::]:32903 has accepted a connection from [localhost]:49180.
[I ProcessGroupNCCL.cpp:686] [Rank 3] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 1, NCCL_ENABLE_TIMING: 1, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: DETAIL, NCCL_DEBUG: INFO, ID=94642481533456
[I ProcessGroupNCCL.cpp:686] [Rank 1] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 1, NCCL_ENABLE_TIMING: 1, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: DETAIL, NCCL_DEBUG: INFO, ID=94638613432384
[I ProcessGroupNCCL.cpp:686] [Rank 2] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 1, NCCL_ENABLE_TIMING: 1, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: DETAIL, NCCL_DEBUG: INFO, ID=94495567931760
[I ProcessGroupNCCL.cpp:686] [Rank 0] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 1, NCCL_ENABLE_TIMING: 1, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: DETAIL, NCCL_DEBUG: INFO, ID=94904165283424
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 4 processes
----------------------------------------------------------------------------------------------------

[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=0, OpType=BROADCAST, TensorShape=[1], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=0, OpType=BROADCAST, TensorShape=[1], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=0, OpType=BROADCAST, TensorShape=[1], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=0, OpType=BROADCAST, TensorShape=[1], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupNCCL.cpp:1340] NCCL_DEBUG: INFO
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=1, OpType=BROADCAST, TensorShape=[61], TensorDtypes=Byte, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=1, OpType=BROADCAST, TensorShape=[61], TensorDtypes=Byte, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=1, OpType=BROADCAST, TensorShape=[61], TensorDtypes=Byte, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=1, OpType=BROADCAST, TensorShape=[61], TensorDtypes=Byte, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=2OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=2OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=2OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=2OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=3, OpType=BROADCAST, TensorShape=[1], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=3, OpType=BROADCAST, TensorShape=[1], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=3, OpType=BROADCAST, TensorShape=[1], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=3, OpType=BROADCAST, TensorShape=[1], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=4, OpType=BROADCAST, TensorShape=[73], TensorDtypes=Byte, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=4, OpType=BROADCAST, TensorShape=[73], TensorDtypes=Byte, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=4, OpType=BROADCAST, TensorShape=[73], TensorDtypes=Byte, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=4, OpType=BROADCAST, TensorShape=[73], TensorDtypes=Byte, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=5OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=5OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=5OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=5OpType=BARRIER)
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
LOCAL_RANK: 3 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
LOCAL_RANK: 2 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=6, OpType=ALLGATHER, TensorShape=[1], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=6, OpType=ALLGATHER, TensorShape=[1], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=6, OpType=ALLGATHER, TensorShape=[1], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=6, OpType=ALLGATHER, TensorShape=[1], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=7, OpType=BROADCAST, TensorShape=[6], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=7, OpType=BROADCAST, TensorShape=[6], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=7, OpType=BROADCAST, TensorShape=[6], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=7, OpType=BROADCAST, TensorShape=[6], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=8, OpType=BROADCAST, TensorShape=[66], TensorDtypes=Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=8, OpType=BROADCAST, TensorShape=[66], TensorDtypes=Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=8, OpType=BROADCAST, TensorShape=[66], TensorDtypes=Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=8, OpType=BROADCAST, TensorShape=[66], TensorDtypes=Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I reducer.cpp:127] Reducer initialized with bucket_bytes_cap: 26214400 first_bucket_bytes_cap: 1048576
[I reducer.cpp:127] Reducer initialized with bucket_bytes_cap: 26214400 first_bucket_bytes_cap: 1048576
[I reducer.cpp:127] Reducer initialized with bucket_bytes_cap: 26214400 first_bucket_bytes_cap: 1048576
[I reducer.cpp:127] Reducer initialized with bucket_bytes_cap: 26214400 first_bucket_bytes_cap: 1048576
[I logger.cpp:215] [Rank 3]: DDP Initialized with: 
broadcast_buffers: 1
bucket_cap_bytes: 26214400
find_unused_parameters: 0
gradient_as_bucket_view: 0
has_sync_bn: 0
is_multi_device_module: 0
iteration: 0
num_parameter_tensors: 2
output_device: 3
rank: 3
total_parameter_size_bytes: 264
world_size: 4
backend_name: nccl
bucket_sizes: 264
cuda_visible_devices: 0,1,2,3
device_ids: 3
dtypes: float
master_addr: 127.0.0.1
master_port: 32903
module_name: SimpleNetwork
nccl_async_error_handling: N/A
nccl_blocking_wait: N/A
nccl_debug: INFO
nccl_ib_timeout: N/A
nccl_nthreads: N/A
nccl_socket_ifname: N/A
torch_distributed_debug: DETAIL

[I logger.cpp:215] [Rank 0]: DDP Initialized with: 
broadcast_buffers: 1
bucket_cap_bytes: 26214400
find_unused_parameters: 0
gradient_as_bucket_view: 0
has_sync_bn: 0
is_multi_device_module: 0
iteration: 0
num_parameter_tensors: 2
output_device: 0
rank: 0
total_parameter_size_bytes: 264
world_size: 4
backend_name: nccl
bucket_sizes: 264
cuda_visible_devices: 0,1,2,3
device_ids: 0
dtypes: float
master_addr: 127.0.0.1
master_port: 32903
module_name: SimpleNetwork
nccl_async_error_handling: N/A
nccl_blocking_wait: N/A
nccl_debug: INFO
nccl_ib_timeout: N/A
nccl_nthreads: N/A
nccl_socket_ifname: N/A
torch_distributed_debug: DETAIL

[I logger.cpp:215] [Rank 1]: DDP Initialized with: 
broadcast_buffers: 1
bucket_cap_bytes: 26214400
find_unused_parameters: 0
gradient_as_bucket_view: 0
has_sync_bn: 0
is_multi_device_module: 0
iteration: 0
num_parameter_tensors: 2
output_device: 1
rank: 1
total_parameter_size_bytes: 264
world_size: 4
backend_name: nccl
bucket_sizes: 264
cuda_visible_devices: 0,1,2,3
device_ids: 1
dtypes: float
master_addr: 127.0.0.1
master_port: 32903
module_name: SimpleNetwork
nccl_async_error_handling: N/A
nccl_blocking_wait: N/A
nccl_debug: INFO
nccl_ib_timeout: N/A
nccl_nthreads: N/A
nccl_socket_ifname: N/A
torch_distributed_debug: DETAIL

[I logger.cpp:215] [Rank 2]: DDP Initialized with: 
broadcast_buffers: 1
bucket_cap_bytes: 26214400
find_unused_parameters: 0
gradient_as_bucket_view: 0
has_sync_bn: 0
is_multi_device_module: 0
iteration: 0
num_parameter_tensors: 2
output_device: 2
rank: 2
total_parameter_size_bytes: 264
world_size: 4
backend_name: nccl
bucket_sizes: 264
cuda_visible_devices: 0,1,2,3
device_ids: 2
dtypes: float
master_addr: 127.0.0.1
master_port: 32903
module_name: SimpleNetwork
nccl_async_error_handling: N/A
nccl_blocking_wait: N/A
nccl_debug: INFO
nccl_ib_timeout: N/A
nccl_nthreads: N/A
nccl_socket_ifname: N/A
torch_distributed_debug: DETAIL

[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=9OpType=BARRIER)

  | Name  | Type   | Params
---------------------------------
0 | layer | Linear | 66    
---------------------------------
66        Trainable params
0         Non-trainable params
66        Total params
0.000     Total estimated model params size (MB)
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=9OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=9OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=9OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=10OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=10OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=10OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=10OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=11OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=11OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=11OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=11OpType=BARRIER)
/myenv/python3.10/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:441: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=4` in the `DataLoader` to improve performance.
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=12OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=12OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=12OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=12OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=13, OpType=ALLREDUCE, TensorShape=[], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=13, OpType=ALLREDUCE, TensorShape=[], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=13, OpType=ALLREDUCE, TensorShape=[], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=13, OpType=ALLREDUCE, TensorShape=[], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=14OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=14OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=14OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=14OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=15, OpType=ALLREDUCE, TensorShape=[], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=15, OpType=ALLREDUCE, TensorShape=[], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=15, OpType=ALLREDUCE, TensorShape=[], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=15, OpType=ALLREDUCE, TensorShape=[], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
/myenv/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py:293: The number of training batches (1) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=16, OpType=ALLREDUCE, TensorShape=[66], TensorDtypes=Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=16, OpType=ALLREDUCE, TensorShape=[66], TensorDtypes=Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=16, OpType=ALLREDUCE, TensorShape=[66], TensorDtypes=Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=16, OpType=ALLREDUCE, TensorShape=[66], TensorDtypes=Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=17OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=17OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=17OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=17OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=18, OpType=ALLREDUCE, TensorShape=[], TensorDtypes=Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=18, OpType=ALLREDUCE, TensorShape=[], TensorDtypes=Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=18, OpType=ALLREDUCE, TensorShape=[], TensorDtypes=Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=18, OpType=ALLREDUCE, TensorShape=[], TensorDtypes=Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=19OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=19OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=19OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=19OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=20, OpType=ALLREDUCE, TensorShape=[], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=20, OpType=ALLREDUCE, TensorShape=[], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=20, OpType=ALLREDUCE, TensorShape=[], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=20, OpType=ALLREDUCE, TensorShape=[], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 3] Running collective: CollectiveFingerPrint(SequenceNumber=21OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 2] Running collective: CollectiveFingerPrint(SequenceNumber=21OpType=BARRIER)
[I ProcessGroupWrapper.cpp:562] [Rank 1] Running collective: CollectiveFingerPrint(SequenceNumber=21, OpType=BROADCAST, TensorShape=[1], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
[I ProcessGroupWrapper.cpp:562] [Rank 0] Running collective: CollectiveFingerPrint(SequenceNumber=21, OpType=BROADCAST, TensorShape=[1], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))
Traceback (most recent call last):
  File "/mydir/examples/nccl_bug.py", line 59, in <module>
    train_network()
  File "/mydir/examples/nccl_bug.py", line 57, in train_network
    trainer.fit(model)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 545, in fit
Traceback (most recent call last):
  File "/mydir/examples/nccl_bug.py", line 59, in <module>
    train_network()
  File "/mydir/examples/nccl_bug.py", line 57, in train_network
    trainer.fit(model)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 545, in fit
Traceback (most recent call last):
  File "/mydir/examples/nccl_bug.py", line 59, in <module>
    train_network()
  File "/mydir/examples/nccl_bug.py", line 57, in train_network
    trainer.fit(model)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 545, in fit
Traceback (most recent call last):
  File "/mydir/examples/nccl_bug.py", line 59, in <module>
    train_network()
  File "/mydir/examples/nccl_bug.py", line 57, in train_network
    trainer.fit(model)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 545, in fit
    call._call_and_handle_interrupt(
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
    call._call_and_handle_interrupt(
    call._call_and_handle_interrupt(
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
    call._call_and_handle_interrupt(
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/myenv/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 102, in launch
  File "/myenv/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 102, in launch
  File "/myenv/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 102, in launch
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/myenv/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 102, in launch
    return function(*args, **kwargs)
    return function(*args, **kwargs)
    return function(*args, **kwargs)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 581, in _fit_impl
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 581, in _fit_impl
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 581, in _fit_impl
    return function(*args, **kwargs)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 581, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
    self._run(model, ckpt_path=ckpt_path)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 990, in _run
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 990, in _run
    self._run(model, ckpt_path=ckpt_path)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 990, in _run
    self._run(model, ckpt_path=ckpt_path)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 990, in _run
    results = self._run_stage()
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1036, in _run_stage
    results = self._run_stage()
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1036, in _run_stage
    results = self._run_stage()
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1036, in _run_stage
    results = self._run_stage()
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1036, in _run_stage
    self.fit_loop.run()
    self.fit_loop.run()
  File "/myenv/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 203, in run
  File "/myenv/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 203, in run
    self.fit_loop.run()
  File "/myenv/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 203, in run
    self.fit_loop.run()
  File "/myenv/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 203, in run
    self.on_advance_end()
  File "/myenv/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 374, in on_advance_end
    self.on_advance_end()
    self.on_advance_end()
  File "/myenv/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 374, in on_advance_end
  File "/myenv/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 374, in on_advance_end
    self.on_advance_end()
  File "/myenv/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 374, in on_advance_end
    call._call_callback_hooks(trainer, "on_train_epoch_end", monitoring_callbacks=True)
    call._call_callback_hooks(trainer, "on_train_epoch_end", monitoring_callbacks=True)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 208, in _call_callback_hooks
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 208, in _call_callback_hooks
    call._call_callback_hooks(trainer, "on_train_epoch_end", monitoring_callbacks=True)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 208, in _call_callback_hooks
    call._call_callback_hooks(trainer, "on_train_epoch_end", monitoring_callbacks=True)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 208, in _call_callback_hooks
    fn(trainer, trainer.lightning_module, *args, **kwargs)
    fn(trainer, trainer.lightning_module, *args, **kwargs)
  File "/myenv/python3.10/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 309, in on_train_epoch_end
  File "/myenv/python3.10/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 311, in on_train_epoch_end
    fn(trainer, trainer.lightning_module, *args, **kwargs)
  File "/myenv/python3.10/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 311, in on_train_epoch_end
    fn(trainer, trainer.lightning_module, *args, **kwargs)
  File "/myenv/python3.10/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 309, in on_train_epoch_end
    monitor_candidates = self._monitor_candidates(trainer)
  File "/myenv/python3.10/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 646, in _monitor_candidates
    self._save_topk_checkpoint(trainer, monitor_candidates)
  File "/myenv/python3.10/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 370, in _save_topk_checkpoint
    monitor_candidates = self._monitor_candidates(trainer)
    self._save_topk_checkpoint(trainer, monitor_candidates)
  File "/myenv/python3.10/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 646, in _monitor_candidates
  File "/myenv/python3.10/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 370, in _save_topk_checkpoint
    monitor_candidates = deepcopy(trainer.callback_metrics)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1634, in callback_metrics
    self._save_none_monitor_checkpoint(trainer, monitor_candidates)
  File "/myenv/python3.10/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 688, in _save_none_monitor_checkpoint
    self._save_none_monitor_checkpoint(trainer, monitor_candidates)
  File "/myenv/python3.10/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 688, in _save_none_monitor_checkpoint
    monitor_candidates = deepcopy(trainer.callback_metrics)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1634, in callback_metrics
    filepath = self._get_metric_interpolated_filepath_name(monitor_candidates, trainer)
  File "/myenv/python3.10/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 639, in _get_metric_interpolated_filepath_name
    return self._logger_connector.callback_metrics
    while self.file_exists(filepath, trainer) and filepath != del_filepath:
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py", line 231, in callback_metrics
  File "/myenv/python3.10/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 752, in file_exists
    filepath = self._get_metric_interpolated_filepath_name(monitor_candidates, trainer)
  File "/myenv/python3.10/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 639, in _get_metric_interpolated_filepath_name
    return trainer.strategy.broadcast(exists)
  File "/myenv/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 298, in broadcast
    return self._logger_connector.callback_metrics
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py", line 231, in callback_metrics
    while self.file_exists(filepath, trainer) and filepath != del_filepath:
  File "/myenv/python3.10/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 752, in file_exists
    metrics = self.metrics["callback"]
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py", line 226, in metrics
    return trainer.strategy.broadcast(exists)
  File "/myenv/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 298, in broadcast
    metrics = self.metrics["callback"]
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py", line 226, in metrics
    torch.distributed.broadcast_object_list(obj, src, group=_group.WORLD)
  File "/myenv/python3.10/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
    return self.trainer._results.metrics(on_step)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py", line 471, in metrics
    return self.trainer._results.metrics(on_step)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py", line 471, in metrics
    torch.distributed.broadcast_object_list(obj, src, group=_group.WORLD)
  File "/myenv/python3.10/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/myenv/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2603, in broadcast_object_list
    return func(*args, **kwargs)
  File "/myenv/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2603, in broadcast_object_list
    value = self._get_cache(result_metric, on_step)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py", line 435, in _get_cache
    value = self._get_cache(result_metric, on_step)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py", line 435, in _get_cache
    result_metric.compute()
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py", line 280, in wrapped_func
    result_metric.compute()
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py", line 280, in wrapped_func
    self._computed = compute(*args, **kwargs)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py", line 243, in compute
    self._computed = compute(*args, **kwargs)
  File "/myenv/python3.10/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py", line 243, in compute
    value = self.meta.sync(self.value.clone())  # `clone` because `sync` is in-place
  File "/myenv/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 330, in reduce
    value = self.meta.sync(self.value.clone())  # `clone` because `sync` is in-place
  File "/myenv/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 330, in reduce
    return _sync_ddp_if_available(tensor, group, reduce_op=reduce_op)
  File "/myenv/python3.10/site-packages/lightning_fabric/utilities/distributed.py", line 171, in _sync_ddp_if_available
    return _sync_ddp_if_available(tensor, group, reduce_op=reduce_op)
  File "/myenv/python3.10/site-packages/lightning_fabric/utilities/distributed.py", line 171, in _sync_ddp_if_available
    broadcast(object_sizes_tensor, src=src, group=group)
  File "/myenv/python3.10/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/myenv/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1906, in broadcast
    return _sync_ddp(result, group=group, reduce_op=reduce_op)
  File "/myenv/python3.10/site-packages/lightning_fabric/utilities/distributed.py", line 220, in _sync_ddp
    return _sync_ddp(result, group=group, reduce_op=reduce_op)
  File "/myenv/python3.10/site-packages/lightning_fabric/utilities/distributed.py", line 220, in _sync_ddp
    broadcast(object_sizes_tensor, src=src, group=group)
  File "/myenv/python3.10/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/myenv/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1906, in broadcast
    torch.distributed.barrier(group=group)
    torch.distributed.barrier(group=group)
  File "/myenv/python3.10/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
  File "/myenv/python3.10/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
    work = default_pg.broadcast([tensor], opts)
RuntimeError: Detected mismatch between collectives on ranks. Rank 1 is running collective: CollectiveFingerPrint(SequenceNumber=21, OpType=BROADCAST, TensorShape=[1], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt))), but Rank 2 is running collective: CollectiveFingerPrint(SequenceNumber=0OpType=REDUCE).Collectives differ in the following aspects: 	 Sequence number: 21vs 0  Op type: BROADCASTvs REDUCE  Tensor Tensor shapes: 1vs   Tensor Tensor dtypes: Longvs   Tensor Tensor devices: TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt))vs 
    return func(*args, **kwargs)
    return func(*args, **kwargs)
  File "/myenv/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3698, in barrier
  File "/myenv/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3698, in barrier
    work = default_pg.broadcast([tensor], opts)
RuntimeError: Detected mismatch between collectives on ranks. Rank 0 is running collective: CollectiveFingerPrint(SequenceNumber=21, OpType=BROADCAST, TensorShape=[1], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt))), but Rank 2 is running collective: CollectiveFingerPrint(SequenceNumber=0OpType=REDUCE).Collectives differ in the following aspects: 	 Sequence number: 21vs 0  Op type: BROADCASTvs REDUCE  Tensor Tensor shapes: 1vs   Tensor Tensor dtypes: Longvs   Tensor Tensor devices: TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt))vs 
    work = group.barrier(opts=opts)
    work = group.barrier(opts=opts)
RuntimeError: Detected mismatch between collectives on ranks. Rank 3 is running collective: CollectiveFingerPrint(SequenceNumber=21OpType=BARRIER), but Rank 0 is running collective: CollectiveFingerPrint(SequenceNumber=0OpType=GATHER).Collectives differ in the following aspects: 	 Sequence number: 21vs 0  Op type: BARRIERvs GATHER
RuntimeError: Detected mismatch between collectives on ranks. Rank 2 is running collective: CollectiveFingerPrint(SequenceNumber=21OpType=BARRIER), but Rank 0 is running collective: CollectiveFingerPrint(SequenceNumber=0OpType=GATHER).Collectives differ in the following aspects: 	 Sequence number: 21vs 0  Op type: BARRIERvs GATHER
[I ProcessGroupNCCL.cpp:874] [Rank 3] Destroyed 1communicators on CUDA device 3
[I ProcessGroupNCCL.cpp:874] [Rank 1] Destroyed 1communicators on CUDA device 1
[I ProcessGroupNCCL.cpp:874] [Rank 0] Destroyed 1communicators on CUDA device 0
[I ProcessGroupNCCL.cpp:874] [Rank 2] Destroyed 1communicators on CUDA device 2

Environment

More info

No response

Start with the log_dict call using sync_dist=True in the provided multi-GPU reproduction, and run it with ranks logging different dictionary keys. Done means the distributed run no longer deadlocks or causes NCCL to die, while keys present across dictionaries are averaged as expected.