deepinsight / deepinsight/insightface
[arcface_torch]“ZeroDivisionError: float division by zero !” When I train on my own dataset
- Dominant language
- Python
- Stars
- 29.7k
- Forks
- 6.1k
- PR merge metrics
- No merged PRs in 30d
Description
Training: 2021-03-31 11:28:15,352-Speed 415.63 samples/sec Loss 36.7039 Epoch: 1 Global Step: 2000 Required: 1 hours
testing verification..
(472, 512)
infer time 1.2834219999999998
Traceback (most recent call last):
File "train.py", line 132, in
main(args_)
File "train.py", line 118, in main
callback_verification(global_step, backbone)
File "/root/arcface_torch/utils/utils_callbacks.py", line 48, in __call__
self.ver_test(backbone, num_update)
File "/root/arcface_torch/utils/utils_callbacks.py", line 27, in ver_test
acc1, std1, acc2, std2, xnorm, embeddings_list = verification.test(
File "/usr/local/lib/python3.8/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/root/arcface_torch/eval/verification.py", line 275, in test
_, _, accuracy, val, val_std, far = evaluate(embeddings, issame_list, nrof_folds=nfolds)
File "/root/arcface_torch/eval/verification.py", line 191, in evaluate
val, val_std, far = calculate_val(thresholds,
File "/root/arcface_torch/eval/verification.py", line 148, in calculate_val
_, far_train[threshold_idx] = calculate_val_far(
File "/root/arcface_torch/eval/verification.py", line 175, in calculate_val_far
far = float(false_accept) / float(n_diff)
ZeroDivisionError: float division by zero
Killing subprocess 3993
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launch.py", line 340, in
main()
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launch.py", line 326, in main
sigkill_handler(signal.SIGTERM, None) # not coming back
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launch.py", line 301, in sigkill_handler
raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-u', 'train.py', '--local_rank=0']' returned non-zero exit status 1.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.