Bug: ValueError: matrix contains invalid numeric entries

INFO 2025-12-04 07:54:24,909 train_utils.py: 268: Train Epoch: [70][200/532] | Batch Time: 0.80 (0.44) | Data Time: 0.00 (0.07) | Mem (GB): 16.00 (16.33/19.00) | Time Elapsed: 00d 04h 24m | Losses/train_all_loss: 2.72e+01 (6.98e+01) | Losses/train_default_loss: 0.00e+00 (0.00e+00)
INFO 2025-12-04 07:54:28,391 train_utils.py: 268: Train Epoch: [70][210/532] | Batch Time: 0.34 (0.44) | Data Time: 0.00 (0.06) | Mem (GB): 16.00 (16.34/19.00) | Time Elapsed: 00d 04h 24m | Losses/train_all_loss: 1.08e+01 (7.20e+01) | Losses/train_default_loss: 0.00e+00 (0.00e+00)
INFO 2025-12-04 07:54:31,771 train_utils.py: 268: Train Epoch: [70][220/532] | Batch Time: 0.34 (0.43) | Data Time: 0.00 (0.06) | Mem (GB): 17.00 (16.33/19.00) | Time Elapsed: 00d 04h 24m | Losses/train_all_loss: 1.47e+02 (7.06e+01) | Losses/train_default_loss: 0.00e+00 (0.00e+00)
INFO 2025-12-04 07:54:35,136 train_utils.py: 268: Train Epoch: [70][230/532] | Batch Time: 0.35 (0.43) | Data Time: 0.00 (0.06) | Mem (GB): 16.00 (16.34/19.00) | Time Elapsed: 00d 04h 25m | Losses/train_all_loss: 5.90e+01 (7.04e+01) | Losses/train_default_loss: 0.00e+00 (0.00e+00)
INFO 2025-12-04 07:54:38,600 train_utils.py: 268: Train Epoch: [70][240/532] | Batch Time: 0.40 (0.43) | Data Time: 0.00 (0.05) | Mem (GB): 16.00 (16.34/19.00) | Time Elapsed: 00d 04h 25m | Losses/train_all_loss: 6.55e+01 (7.00e+01) | Losses/train_default_loss: 0.00e+00 (0.00e+00)
[rank0]: Traceback (most recent call last):
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3/train/train.py", line 339, in
[rank0]: main(args)
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3/train/train.py", line 310, in main
[rank0]: single_node_runner(cfg, main_port)
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3/train/train.py", line 71, in single_node_runner
[rank0]: single_proc_run(local_rank=0, main_port=main_port, cfg=cfg, world_size=num_proc)
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3/train/train.py", line 58, in single_proc_run
[rank0]: trainer.run()
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3/train/trainer.py", line 567, in run
[rank0]: self.run_train()
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3/train/trainer.py", line 588, in run_train
[rank0]: outs = self.train_epoch(dataloader)
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3/train/trainer.py", line 809, in train_epoch
[rank0]: self._run_step(batch, phase, loss_mts, extra_loss_mts)
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3/train/trainer.py", line 946, in _run_step
[rank0]: loss_dict, batch_size, extra_losses = self._step(
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3/train/trainer.py", line 501, in _step
[rank0]: find_stages = model(batch)
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3_venv/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1661, in forward
[rank0]: else self._run_ddp_forward(*inputs, **kwargs)
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3_venv/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1487, in _run_ddp_forward
[rank0]: return self.module(*inputs, **kwargs) # type: ignore[index]
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3/model/sam3_image.py", line 567, in forward
[rank0]: out = self.forward_grounding(
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3/model/sam3_image.py", line 492, in forward_grounding
[rank0]: self._compute_matching(out, self.back_convert(find_target))
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3/model/sam3_image.py", line 579, in _compute_matching
[rank0]: out["indices"] = self.matcher(out, targets)
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3_venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3/train/matcher.py", line 643, in forward
[rank0]: indices = [
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3/train/matcher.py", line 644, in
[rank0]: _do_matching(c, repeats=repeats, do_filtering=do_filtering)
[rank0]: File "/home/qmask_quangnh58/detect/sam3/sam3/train/matcher.py", line 19, in _do_matching
[rank0]: i, j = linear_sum_assignment(cost)
[rank0]: ValueError: matrix contains invalid numeric entries

Start with sam3/train/matcher.py at _do_matching and trace the matcher call from sam3/model/sam3_image.py::_compute_matching, then follow the training path through sam3/train/trainer.py. Reproduce the reported failure using the training setup and inspect the cost matrix values before linear_sum_assignment. Done means the invalid numeric entries are explained and training completes the affected matching step without this error.

Written by the indexing model from the issue text.