The following issue occurred while fine-tuning SAM2.

WARNING:root:Loading failed (id=178); Retry 42 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000178_10'
WARNING:root:Loading failed (id=121); Retry 43 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000121_10'
WARNING:root:Loading failed (id=102); Retry 44 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000102_10'
WARNING:root:Loading failed (id=191); Retry 45 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000191_10'
WARNING:root:Loading failed (id=189); Retry 46 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000189_10'
ERROR 2024-10-18 16:36:21,929 sam2_datasets.py: 63: Caught UnboundLocalError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/guo_jingzhu/miniconda3/envs/sam2/lib/python3.12/site-packages/torch/utils/data/_utils/worker.py", line 351, in _worker_loop
data = fetcher.fetch(index) # type: ignore[possibly-undefined]
^^^^^^^^^^^^^^^^^^^^
File "/home/guo_jingzhu/miniconda3/envs/sam2/lib/python3.12/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
~~~~~~~~~~~~^^^^^
File "/home/guo_jingzhu/sam2/training/dataset/vos_dataset.py", line 132, in __getitem__
return self._get_datapoint(idx)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/guo_jingzhu/sam2/training/dataset/vos_dataset.py", line 74, in _get_datapoint
datapoint = self.construct(video, sampled_frms_and_objs, segment_loader)
^^^^^
UnboundLocalError: cannot access local variable 'video' where it is not associated with a value

WARNING:root:Loading failed (id=132); Retry 47 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000132_10'
WARNING:root:Loading failed (id=47); Retry 48 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000047_10'
WARNING:root:Loading failed (id=167); Retry 49 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000167_10'
WARNING:root:Loading failed (id=9); Retry 50 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000009_10'
WARNING:root:Loading failed (id=142); Retry 51 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000142_10'
WARNING:root:Loading failed (id=7); Retry 52 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000007_10'
WARNING:root:Loading failed (id=157); Retry 0 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000157_10'
WARNING:root:Loading failed (id=72); Retry 1 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000072_10'
WARNING:root:Loading failed (id=1); Retry 2 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000001_10'
WARNING:root:Loading failed (id=164); Retry 3 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000164_10'
WARNING:root:Loading failed (id=58); Retry 4 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000058_10'
[rank0]: Traceback (most recent call last):
[rank0]: File "/home/guo_jingzhu/sam2/training/train.py", line 287, in
[rank0]: main(args)
[rank0]: File "/home/guo_jingzhu/sam2/training/train.py", line 256, in main
[rank0]: single_node_runner(cfg, main_port)
[rank0]: File "/home/guo_jingzhu/sam2/training/train.py", line 53, in single_node_runner
[rank0]: single_proc_run(local_rank=0, main_port=main_port, cfg=cfg, world_size=num_proc)
[rank0]: File "/home/guo_jingzhu/sam2/training/train.py", line 41, in single_proc_run
[rank0]: trainer.run()
[rank0]: File "/home/guo_jingzhu/sam2/training/trainer.py", line 515, in run
[rank0]: self.run_train()
[rank0]: File "/home/guo_jingzhu/sam2/training/trainer.py", line 532, in run_train
[rank0]: outs = self.train_epoch(dataloader)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/guo_jingzhu/sam2/training/trainer.py", line 740, in train_epoch
[rank0]: for data_iter, batch in enumerate(train_loader):
[rank0]: File "/home/guo_jingzhu/sam2/training/dataset/sam2_datasets.py", line 64, in __next__
[rank0]: raise e
[rank0]: File "/home/guo_jingzhu/sam2/training/dataset/sam2_datasets.py", line 56, in __next__
[rank0]: item = next(self._iter_dls[dataset_idx])
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/guo_jingzhu/miniconda3/envs/sam2/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 701, in __next__
[rank0]: data = self._next_data()
[rank0]: ^^^^^^^^^^^^^^^^^
[rank0]: File "/home/guo_jingzhu/miniconda3/envs/sam2/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 1465, in _next_data
[rank0]: return self._process_data(data)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/guo_jingzhu/miniconda3/envs/sam2/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 1491, in _process_data
[rank0]: data.reraise()
[rank0]: File "/home/guo_jingzhu/miniconda3/envs/sam2/lib/python3.12/site-packages/torch/_utils.py", line 715, in reraise
[rank0]: raise exception
[rank0]: UnboundLocalError: Caught UnboundLocalError in DataLoader worker process 0.
[rank0]: Original Traceback (most recent call last):
[rank0]: File "/home/guo_jingzhu/miniconda3/envs/sam2/lib/python3.12/site-packages/torch/utils/data/_utils/worker.py", line 351, in _worker_loop
[rank0]: data = fetcher.fetch(index) # type: ignore[possibly-undefined]
[rank0]: ^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/guo_jingzhu/miniconda3/envs/sam2/lib/python3.12/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
[rank0]: data = [self.dataset[idx] for idx in possibly_batched_index]
[rank0]: ~~~~~~~~~~~~^^^^^
[rank0]: File "/home/guo_jingzhu/sam2/training/dataset/vos_dataset.py", line 132, in __getitem__
[rank0]: return self._get_datapoint(idx)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/guo_jingzhu/sam2/training/dataset/vos_dataset.py", line 74, in _get_datapoint
[rank0]: datapoint = self.construct(video, sampled_frms_and_objs, segment_loader)
[rank0]: ^^^^^
[rank0]: UnboundLocalError: cannot access local variable 'video' where it is not associated with a value

WARNING:root:Loading failed (id=112); Retry 5 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000112_10'
WARNING:root:Loading failed (id=180); Retry 6 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000180_10'
WARNING:root:Loading failed (id=32); Retry 7 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000032_10'
WARNING:root:Loading failed (id=65); Retry 8 with exception: [Errno 2] No such file or directory: 'kitti/training/semantic_rgb/000065_10'
[rank0]:[W1018 16:36:22.013044251 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())

I don't know why this issue occurred. I have also modified vos_dataset.py and vos_segment_loader.py. This is my first time dealing with segmentation problems, and I hope someone can tell me what the structure of the SA1B dataset format is.