deepinsight / deepinsight/insightface

training stop unexpectedly

Open
#1,196 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
29.7k
Forks
6.1k
PR merge metrics
No merged PRs in 30d

Description

Hi, I prepared the data for training insightface as described in the [guide](https://github.com/Talgin/preparing_data) of @Talgin . However, when the training start, this following error occured:

> **$ CUDA_VISIBLE_DEVICES='' python -u train.py --network r100 --loss arcface --dataset ktnv**
use cpu
prefix ./models/r100-arcface-ktnv/model
image_size [112, 112]
num_classes 37
Called with argument: Namespace(batch_size=128, ckpt=3, ctx_num=1, dataset='ktnv', frequent=20, image_channel=3, kvstore='device', loss='arcface', lr=0.1, lr_steps='100000,160000,220000', models_root='./models', mom=0.9, network='r100', per_batch_size=128, pretrained='', pretrained_epoch=1, rescale_threshold=0, verbose=2000, wd=0.0005) {'bn_mom': 0.9, 'workspace': 256, 'emb_size': 512, 'ckpt_embedding': True, 'net_se': 0, 'net_act': 'prelu', 'net_unit': 3, 'net_input': 1, 'net_blocks': [1, 4, 6, 2], 'net_output': 'E', 'net_multiplier': 1.0, 'val_targets': ['lfw'], 'ce_loss': True, 'fc7_lr_mult': 1.0, 'fc7_wd_mult': 1.0, 'fc7_no_bias': False, 'max_steps': 0, 'data_rand_mirror': True, 'data_cutoff': False, 'data_color': 0, 'data_images_filter': 0, 'count_flops': True, 'memonger': False, 'loss_name': 'margin_softmax', 'loss_s': 64.0, 'loss_m1': 1.0, 'loss_m2': 0.5, 'loss_m3': 0.0, 'net_name': 'fresnet', 'num_layers': 100, 'dataset': 'ktnv', 'dataset_path': '../datasets/ktnv', 'num_classes': 37, 'image_shape': [112, 112, 3], 'loss': 'arcface', 'network': 'r100', 'num_workers': 1, 'batch_size': 128, 'per_batch_size': 128}
0 1 E 3 prelu False
Network FLOPs: 24.2G
INFO:root:loading recordio ../datasets/ktnv/train.rec...
header0 label [38. 42.]
id2range 4
37
rand_mirror True
loading bin 0
(456, 3, 112, 112)
ver lfw
lr_steps [100000, 160000, 220000]
call reset()
/home/extreme45nm/anaconda3/lib/python3.7/site-packages/mxnet/module/base_module.py:504: UserWarning: Optimizer created manually outside Module but rescale_grad is not normalized to 1.0/batch_size/num_workers (1.0 vs. 0.0078125). Is this intended?
optimizer_params=optimizer_params)
Traceback (most recent call last):
File "train.py", line 379, in
main()
File "train.py", line 376, in main
train_net(args)
File "train.py", line 370, in train_net
epoch_end_callback = epoch_cb )
File "/home/extreme45nm/anaconda3/lib/python3.7/site-packages/mxnet/module/base_module.py", line 520, in fit
next_data_batch = next(data_iter)
File "/home/extreme45nm/anaconda3/lib/python3.7/site-packages/mxnet/io/io.py", line 230, in next
return self.next()
File "/home/extreme45nm/anaconda3/lib/python3.7/site-packages/mxnet/io/io.py", line 476, in next
raise StopIteration
StopIteration

I am stuck at this point, please give me some hint to solve this.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.