Loss is nan

How to solve:
> nan/inf value in loss/matcher

And why:

> Warning, empty mask found, approximating from box?

```bash
INFO 2026-01-30 10:09:22,829 train_utils.py: 269: Train Epoch: [0][2610/3783] | Batch Time: 0.44 (0.50) | Data Time: 0.00 (0.03) | Mem (GB): 7.00 (7.06/10.00) | Time Elapsed: 00d 00h 22m | Losses/train_all_loss: 1.75e+02 (1.92e+02) | Losses/train_default_loss: 0.00e+00 (0.00e+00)
Warning, empty mask found, approximating from box
INFO 2026-01-30 10:09:27,812 train_utils.py: 269: Train Epoch: [0][2620/3783] | Batch Time: 0.45 (0.50) | Data Time: 0.00 (0.03) | Mem (GB): 7.00 (7.05/10.00) | Time Elapsed: 00d 00h 22m | Losses/train_all_loss: 3.17e+02 (1.92e+02) | Losses/train_default_loss: 0.00e+00 (0.00e+00)
INFO 2026-01-30 10:09:32,290 train_utils.py: 269: Train Epoch: [0][2630/3783] | Batch Time: 0.43 (0.50) | Data Time: 0.00 (0.03) | Mem (GB): 7.00 (7.06/10.00) | Time Elapsed: 00d 00h 22m | Losses/train_all_loss: 9.73e+01 (1.92e+02) | Losses/train_default_loss: 0.00e+00 (0.00e+00)
/*/*/sam3/sam3/train/matcher.py:26: UserWarning: Found 1600 NaN and 0 Inf values in cost matrix. Replacing with 1e8.
warnings.warn(f"Found {num_nan} NaN and {num_inf} Inf values in cost matrix. Replacing with 1e8.")
/*/*/sam3/sam3/train/matcher.py:26: UserWarning: Found 400 NaN and 0 Inf values in cost matrix. Replacing with 1e8.
warnings.warn(f"Found {num_nan} NaN and {num_inf} Inf values in cost matrix. Replacing with 1e8.")
ERROR 2026-01-30 10:09:34,300 trainer.py: 953: Loss is nan, attempting to stop training
terminate called without an active exception
Exception ignored in:
Traceback (most recent call last):
File "/opt/software/anaconda3/envs/*/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 1663, in __del__
self._shutdown_workers()
File "/opt/software/anaconda3/envs/*/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 1627, in _shutdown_workers
w.join(timeout=_utils.MP_STATUS_CHECK_INTERVAL)
File "/opt/software/anaconda3/envs/*/lib/python3.12/multiprocessing/process.py", line 149, in join
res = self._popen.wait(timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/software/anaconda3/envs/*/lib/python3.12/multiprocessing/popen_fork.py", line 40, in wait
if not wait([self.sentinel], timeout):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/software/anaconda3/envs/*/lib/python3.12/multiprocessing/connection.py", line 1136, in wait
ready = selector.select(timeout)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/software/anaconda3/envs/*/lib/python3.12/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/software/anaconda3/envs/*/lib/python3.12/site-packages/torch/utils/data/_utils/signal_handling.py", line 73, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 4102965) is killed by signal: Aborted.
[rank0]: Traceback (most recent call last):
[rank0]: File "/*/*/sam3/sam3/train/train.py", line 338, in
[rank0]: main(args)
[rank0]: File "/*/*/sam3/sam3/train/train.py", line 309, in main
[rank0]: single_node_runner(cfg, main_port)
[rank0]: File "/*/*/sam3/sam3/train/train.py", line 70, in single_node_runner
[rank0]: single_proc_run(local_rank=0, main_port=main_port, cfg=cfg, world_size=num_proc)
[rank0]: File "/*/*/sam3/sam3/train/train.py", line 57, in single_proc_run
[rank0]: trainer.run()
[rank0]: File "/*/*/sam3/sam3/train/trainer.py", line 563, in run
[rank0]: self.run_train()
[rank0]: File "/*/*/sam3/sam3/train/trainer.py", line 584, in run_train
[rank0]: outs = self.train_epoch(dataloader)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/*/*/sam3/sam3/train/trainer.py", line 873, in train_epoch
[rank0]: raise e
[rank0]: File "/*/*/sam3/sam3/train/trainer.py", line 805, in train_epoch
[rank0]: self._run_step(batch, phase, loss_mts, extra_loss_mts)
[rank0]: File "/*/*/sam3/sam3/train/trainer.py", line 955, in _run_step
[rank0]: raise FloatingPointError(error_msg)
[rank0]: FloatingPointError: Loss is nan, attempting to stop training
[rank0]:[W130 10:09:37.999099613 ProcessGroupNCCL.cpp:1476] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
```

Start with sam3/train/matcher.py where NaN costs are reported, then follow the training path through sam3/train/trainer.py and sam3/train/train.py. Reproduce the failure around the empty-mask warning and inspect the batch or configuration that triggers it. Done means the cause is identified and training no longer reaches the reported NaN loss, with a regression check if the repository provides one.

Written by the indexing model from the issue text.