facebookresearch / facebookresearch/detectron2

IndexError

Open
#5,012 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34.7k
Forks
7.9k
PR merge metrics
No merged PRs in 30d

Description

I have repeatedly verified my dataset and found no issues (including attempting to start with category id from 0 or 1). However, there is still an issue of index overflow in the dataloader.

## Instructions To Reproduce the Issue:

Full runnable code or full changes you made:
I only registered my own Coco type dataset during training.
```
register_coco_instances("coco_my_train", {}, TRAIN_JSON, TRAIN_PATH)
register_coco_instances("coco_my_val", {}, VAL_JSON, VAL_PATH)
```

__Full logs__ or other relevant observations:
```
Traceback (most recent call last):
File ".../engine/train_loop.py", line 134, in train
self.run_step()
File ".../engine/defaults.py", line 429, in run_step
self._trainer.run_step()
File ".../engine/train_loop.py", line 222, in run_step
data = next(self._data_loader_iter)
File ".../data/common.py", line 179, in __iter__
for d in self.dataset:
File ".../python3.9/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
data = self._next_data()
File ".../python3.9/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
return self._process_data(data)
File ".../python3.9/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
data.reraise()
File ".../python3.9/site-packages/torch/_utils.py", line 543, in reraise
raise exception
IndexError: Caught IndexError in DataLoader worker process 0.
Original Traceback (most recent call last):
File ".../python3.9/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
data = fetcher.fetch(index)
File ".../python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File ".../python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 58, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File ".../data/common.py", line 43, in __getitem__
data = self._map_func(self._dataset[cur_idx])
File ".../data/common.py", line 107, in __getitem__
start_addr = 0 if idx == 0 else self._addr[idx - 1].item()
IndexError: index 102724527 is out of bounds for axis 0 with size 15000
```

## Attempted Solution:
After multiple attempts, it was found that setting num_workers to 1 can start training normally (although there may not be any problems later), but setting num_workers to greater than 1 will result in errors in the previous iterations.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.