deepinsight / deepinsight/insightface

Distributed validation fails

Open
#1,999 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
29.7k
Forks
6.1k
PR merge metrics
No merged PRs in 30d

Description

Hello everyone,
I am facing an issue with `train.py` when `--launcher` is set to `pytorch` the validation step fails with the following stack trace:
```python
File "test.py", line 208, in
main()
File "test.py", line 185, in main
args.gpu_collect)
File "/home/blablabla/task3/scrfd/detection/scrfd/mmdet/apis/test.py", line 97, in multi_gpu_test
result = model(return_loss=False, rescale=True, **data)
File "/home/blablabla/anaconda3/envs/scrfd/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/blablabla/anaconda3/envs/scrfd/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 886, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/home/blablabla/anaconda3/envs/scrfd/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/blablabla/task3/scrfd/detection/scrfd/mmcv/mmcv/runner/fp16_utils.py", line 84, in new_func
return old_func(*args, **kwargs)
File "/home/blablabla/task3/scrfd/detection/scrfd/mmdet/models/detectors/base.py", line 182, in forward
return self.forward_test(img, img_metas, **kwargs)
File "/home/blablabla/task3/scrfd/detection/scrfd/mmdet/models/detectors/base.py", line 149, in forward_test
img_meta[img_id]['batch_input_shape'] = tuple(img.size()[-2:])
TypeError: 'DataContainer' object is not subscriptable
```
I used `test.py` for debugging to save the training time. After some debugging, I noticed that `forward_test()` in [mmdet/models/detectors/base.py](https://github.com/deepinsight/insightface/blob/b5958bc1d81988dbcc113ababdfdc82125a0717a/detection/scrfd/mmdet/models/detectors/base.py#L130) expects `img_metas` to be `(List[List[dict]])` but under distributed training it's a `(List[DataContainer(List[List[dict()]])])` check below

```python
print('\ntype img_metas:', type(img_metas))
print('type img_metas[0]:', type(img_metas[0]))
print('type img_metas[0].data:', type(img_metas[0].data))
print('type img_metas[0].data[0]:', type(img_metas[0].data[0]))
print('type img_metas[0].data[0][0]:', type(img_metas[0].data[0][0]))
exit()
#Output:
type img_metas:
type img_metas[0]:
type img_metas[0].data:
type img_metas[0].data[0]:
type img_metas[0].data[0][0]:
```
Relaunching with `--launcher` set to `none` works normally. Also modifying the code to deal with the `DataContainer` works but is there a solution that works for both distributed and non distributed flow?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.