deepinsight / deepinsight/insightface
Distributed validation fails
- Dominant language
- Python
- Stars
- 29.7k
- Forks
- 6.1k
- PR merge metrics
- No merged PRs in 30d
Description
Hello everyone,
I am facing an issue with `train.py` when `--launcher` is set to `pytorch` the validation step fails with the following stack trace:
```python
File "test.py", line 208, in
main()
File "test.py", line 185, in main
args.gpu_collect)
File "/home/blablabla/task3/scrfd/detection/scrfd/mmdet/apis/test.py", line 97, in multi_gpu_test
result = model(return_loss=False, rescale=True, **data)
File "/home/blablabla/anaconda3/envs/scrfd/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/blablabla/anaconda3/envs/scrfd/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 886, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/home/blablabla/anaconda3/envs/scrfd/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/blablabla/task3/scrfd/detection/scrfd/mmcv/mmcv/runner/fp16_utils.py", line 84, in new_func
return old_func(*args, **kwargs)
File "/home/blablabla/task3/scrfd/detection/scrfd/mmdet/models/detectors/base.py", line 182, in forward
return self.forward_test(img, img_metas, **kwargs)
File "/home/blablabla/task3/scrfd/detection/scrfd/mmdet/models/detectors/base.py", line 149, in forward_test
img_meta[img_id]['batch_input_shape'] = tuple(img.size()[-2:])
TypeError: 'DataContainer' object is not subscriptable
```
I used `test.py` for debugging to save the training time. After some debugging, I noticed that `forward_test()` in [mmdet/models/detectors/base.py](https://github.com/deepinsight/insightface/blob/b5958bc1d81988dbcc113ababdfdc82125a0717a/detection/scrfd/mmdet/models/detectors/base.py#L130) expects `img_metas` to be `(List[List[dict]])` but under distributed training it's a `(List[DataContainer(List[List[dict()]])])` check below
```python
print('\ntype img_metas:', type(img_metas))
print('type img_metas[0]:', type(img_metas[0]))
print('type img_metas[0].data:', type(img_metas[0].data))
print('type img_metas[0].data[0]:', type(img_metas[0].data[0]))
print('type img_metas[0].data[0][0]:', type(img_metas[0].data[0][0]))
exit()
#Output:
type img_metas:
type img_metas[0]:
type img_metas[0].data:
type img_metas[0].data[0]:
type img_metas[0].data[0][0]:
```
Relaunching with `--launcher` set to `none` works normally. Also modifying the code to deal with the `DataContainer` works but is there a solution that works for both distributed and non distributed flow?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.