Based on FP16, the network training of resnet50 was carried out, and the existence accuracy randomly converged to 0.76

Using the inagenet dataset, on the resnet50 network, based on FP16 training with the model-2.11.0 code, the network has random convergence, is this phenomenon known, is there any solution at present? Is there any good way to debug this problem? Thanks.

I0717 10:42:45.093785 47025319389696 controller.py:291] eval | step: 10009 | eval time: 74.7 sec | output: {'test_accuracy': 0.08882, 'test_loss': 1.2578498}
I0717 11:19:43.333361 47025319389696 controller.py:291] eval | step: 20018 | eval time: 47.9 sec | output: {'test_accuracy': 0.18752, 'test_loss': 1.0495228}
I0717 11:56:49.300491 47025319389696 controller.py:291] eval | step: 30027 | eval time: 48.0 sec | output: {'test_accuracy': 0.2584, 'test_loss': 0.9102226}
I0717 12:33:43.809209 47025319389696 controller.py:291] eval | step: 40036 | eval time: 47.3 sec | output: {'test_accuracy': 0.33168, 'test_loss': 0.77455497}
I0717 13:10:37.399015 47025319389696 controller.py:291] eval | step: 50045 | eval time: 48.6 sec | output: {'test_accuracy': 0.32974, 'test_loss': 0.7866249}
I0717 13:47:31.637742 47025319389696 controller.py:291] eval | step: 60054 | eval time: 48.0 sec | output: {'test_accuracy': 0.32832, 'test_loss': 0.7928696}
I0717 14:24:25.244748 47025319389696 controller.py:291] eval | step: 70063 | eval time: 47.4 sec | output: {'test_accuracy': 0.36952, 'test_loss': 0.7357325}
I0717 15:01:26.831945 47025319389696 controller.py:291] eval | step: 80072 | eval time: 49.6 sec | output: {'test_accuracy': 0.33716, 'test_loss': 0.7839137}
I0717 15:39:10.716763 47025319389696 controller.py:291] eval | step: 90081 | eval time: 51.2 sec | output: {'test_accuracy': 0.32314, 'test_loss': 0.82485163}
I0717 16:17:14.804115 47025319389696 controller.py:291] eval | step: 100090 | eval time: 49.1 sec | output: {'test_accuracy': 0.36102, 'test_loss': 0.7599265}
I0717 16:54:38.272225 47025319389696 controller.py:291] eval | step: 110099 | eval time: 49.5 sec | output: {'test_accuracy': 0.42732, 'test_loss': 0.6477533}
I0717 17:32:14.305328 47025319389696 controller.py:291] eval | step: 120108 | eval time: 49.1 sec | output: {'test_accuracy': 0.3769, 'test_loss': 0.745124}
I0717 18:19:19.995545 47025319389696 controller.py:291] eval | step: 130117 | eval time: 196.2 sec | output: {'test_accuracy': 0.43222, 'test_loss': 0.63742775}
I0717 19:01:38.542571 47025319389696 controller.py:291] eval | step: 140126 | eval time: 49.1 sec | output: {'test_accuracy': 0.43536, 'test_loss': 0.63410264}
I0717 19:39:52.303068 47025319389696 controller.py:291] eval | step: 150135 | eval time: 46.7 sec | output: {'test_accuracy': 0.43782, 'test_loss': 0.6309778}
I0717 20:17:52.557711 47025319389696 controller.py:291] eval | step: 160144 | eval time: 49.5 sec | output: {'test_accuracy': 0.44036, 'test_loss': 0.6212317}
I0717 20:55:54.236690 47025319389696 controller.py:291] eval | step: 170153 | eval time: 47.7 sec | output: {'test_accuracy': 0.4211, 'test_loss': 0.6638591}
I0717 21:33:46.721763 47025319389696 controller.py:291] eval | step: 180162 | eval time: 48.7 sec | output: {'test_accuracy': 0.41044, 'test_loss': 0.6806779}
I0717 22:11:56.432586 47025319389696 controller.py:291] eval | step: 190171 | eval time: 48.9 sec | output: {'test_accuracy': 0.4227, 'test_loss': 0.6592407}
I0717 22:49:48.895009 47025319389696 controller.py:291] eval | step: 200180 | eval time: 47.1 sec | output: {'test_accuracy': 0.41614, 'test_loss': 0.6689654}
I0717 23:27:56.641305 47025319389696 controller.py:291] eval | step: 210189 | eval time: 49.0 sec | output: {'test_accuracy': 0.43086, 'test_loss': 0.6471556}
I0718 00:06:08.226011 47025319389696 controller.py:291] eval | step: 220198 | eval time: 49.0 sec | output: {'test_accuracy': 0.43288, 'test_loss': 0.651372}
I0718 00:43:44.846733 47025319389696 controller.py:291] eval | step: 230207 | eval time: 48.6 sec | output: {'test_accuracy': 0.44876, 'test_loss': 0.61514467}
I0718 01:21:01.860804 47025319389696 controller.py:291] eval | step: 240216 | eval time: 47.8 sec | output: {'test_accuracy': 0.39994, 'test_loss': 0.6896438}
I0718 01:58:26.185002 47025319389696 controller.py:291] eval | step: 250225 | eval time: 49.3 sec | output: {'test_accuracy': 0.43974, 'test_loss': 0.63887787}
I0718 02:35:51.395891 47025319389696 controller.py:291] eval | step: 260234 | eval time: 48.7 sec | output: {'test_accuracy': 0.42148, 'test_loss': 0.65387714}
I0718 03:12:58.957236 47025319389696 controller.py:291] eval | step: 270243 | eval time: 48.0 sec | output: {'test_accuracy': 0.42988, 'test_loss': 0.64366835}
I0718 03:50:05.619722 47025319389696 controller.py:291] eval | step: 280252 | eval time: 49.1 sec | output: {'test_accuracy': 0.4292, 'test_loss': 0.64544094}
I0718 04:27:19.267962 47025319389696 controller.py:291] eval | step: 290261 | eval time: 47.7 sec | output: {'test_accuracy': 0.43688, 'test_loss': 0.633886}
I0718 05:04:16.388976 47025319389696 controller.py:291] eval | step: 300270 | eval time: 47.0 sec | output: {'test_accuracy': 0.4378, 'test_loss': 0.6427796}
I0718 05:41:28.633920 47025319389696 controller.py:291] eval | step: 310279 | eval time: 47.1 sec | output: {'test_accuracy': 0.60604, 'test_loss': 0.41581097}
I0718 06:18:23.006313 47025319389696 controller.py:291] eval | step: 320288 | eval time: 47.6 sec | output: {'test_accuracy': 0.60958, 'test_loss': 0.41247183}
I0718 06:55:19.940962 47025319389696 controller.py:291] eval | step: 330297 | eval time: 47.5 sec | output: {'test_accuracy': 0.62418, 'test_loss': 0.3925642}
I0718 07:32:11.406366 47025319389696 controller.py:291] eval | step: 340306 | eval time: 49.3 sec | output: {'test_accuracy': 0.60878, 'test_loss': 0.41140762}
I0718 08:09:13.098378 47025319389696 controller.py:291] eval | step: 350315 | eval time: 46.3 sec | output: {'test_accuracy': 0.61416, 'test_loss': 0.40241298}
I0718 08:46:11.470759 47025319389696 controller.py:291] eval | step: 360324 | eval time: 48.7 sec | output: {'test_accuracy': 0.61666, 'test_loss': 0.40103617}
I0718 09:23:19.292029 47025319389696 controller.py:291] eval | step: 370333 | eval time: 47.6 sec | output: {'test_accuracy': 0.64238, 'test_loss': 0.3717267}
I0718 09:59:58.652599 47025319389696 controller.py:291] eval | step: 380342 | eval time: 46.9 sec | output: {'test_accuracy': 0.63504, 'test_loss': 0.37905875}
I0718 10:36:30.269412 47025319389696 controller.py:291] eval | step: 390351 | eval time: 46.7 sec | output: {'test_accuracy': 0.62588, 'test_loss': 0.38983408}
I0718 11:12:59.710509 47025319389696 controller.py:291] eval | step: 400360 | eval time: 47.5 sec | output: {'test_accuracy': 0.62752, 'test_loss': 0.39042756}
I0718 11:49:34.702973 47025319389696 controller.py:291] eval | step: 410369 | eval time: 47.6 sec | output: {'test_accuracy': 0.63086, 'test_loss': 0.38619643}
I0718 12:26:10.658050 47025319389696 controller.py:291] eval | step: 420378 | eval time: 46.4 sec | output: {'test_accuracy': 0.61492, 'test_loss': 0.40776753}
I0718 13:02:54.373109 47025319389696 controller.py:291] eval | step: 430387 | eval time: 47.1 sec | output: {'test_accuracy': 0.61482, 'test_loss': 0.40698445}
I0718 13:39:56.424064 47025319389696 controller.py:291] eval | step: 440396 | eval time: 46.6 sec | output: {'test_accuracy': 0.61064, 'test_loss': 0.40885204}
I0718 14:16:56.972168 47025319389696 controller.py:291] eval | step: 450405 | eval time: 47.2 sec | output: {'test_accuracy': 0.6101, 'test_loss': 0.41232613}
I0718 14:53:49.710101 47025319389696 controller.py:291] eval | step: 460414 | eval time: 50.2 sec | output: {'test_accuracy': 0.62022, 'test_loss': 0.39717302}
I0718 15:31:00.055065 47025319389696 controller.py:291] eval | step: 470423 | eval time: 47.8 sec | output: {'test_accuracy': 0.63738, 'test_loss': 0.3755849}
I0718 16:26:43.115331 47025319389696 controller.py:291] eval | step: 480432 | eval time: 48.6 sec | output: {'test_accuracy': 0.63156, 'test_loss': 0.38420147}
I0718 17:14:57.232034 47025319389696 controller.py:291] eval | step: 490441 | eval time: 47.9 sec | output: {'test_accuracy': 0.61762, 'test_loss': 0.40265194}
I0718 18:08:04.532505 47025319389696 controller.py:291] eval | step: 500450 | eval time: 47.4 sec | output: {'test_accuracy': 0.63422, 'test_loss': 0.3809521}
I0718 18:45:18.862020 47025319389696 controller.py:291] eval | step: 510459 | eval time: 50.4 sec | output: {'test_accuracy': 0.62098, 'test_loss': 0.39487627}
I0718 19:22:18.786072 47025319389696 controller.py:291] eval | step: 520468 | eval time: 47.2 sec | output: {'test_accuracy': 0.61512, 'test_loss': 0.40484685}
I0718 19:59:17.472421 47025319389696 controller.py:291] eval | step: 530477 | eval time: 47.3 sec | output: {'test_accuracy': 0.63776, 'test_loss': 0.37440324}
I0718 20:46:59.059379 47025319389696 controller.py:291] eval | step: 540486 | eval time: 203.9 sec | output: {'test_accuracy': 0.62722, 'test_loss': 0.39110208}
I0718 21:45:11.991321 47025319389696 controller.py:291] eval | step: 550495 | eval time: 47.9 sec | output: {'test_accuracy': 0.63562, 'test_loss': 0.38335004}
I0718 22:22:17.513960 47025319389696 controller.py:291] eval | step: 560504 | eval time: 46.5 sec | output: {'test_accuracy': 0.61826, 'test_loss': 0.40100682}
I0718 22:59:05.393987 47025319389696 controller.py:291] eval | step: 570513 | eval time: 46.1 sec | output: {'test_accuracy': 0.63496, 'test_loss': 0.3817627}
I0718 23:36:01.059101 47025319389696 controller.py:291] eval | step: 580522 | eval time: 47.2 sec | output: {'test_accuracy': 0.61354, 'test_loss': 0.40860355}
I0719 00:12:57.767053 47025319389696 controller.py:291] eval | step: 590531 | eval time: 48.7 sec | output: {'test_accuracy': 0.62766, 'test_loss': 0.3902482}
I0719 00:50:08.519172 47025319389696 controller.py:291] eval | step: 600540 | eval time: 47.3 sec | output: {'test_accuracy': 0.62992, 'test_loss': 0.38722458}
I0719 01:27:14.241234 47025319389696 controller.py:291] eval | step: 610549 | eval time: 49.5 sec | output: {'test_accuracy': 0.6785, 'test_loss': 0.3319877}
I0719 02:04:12.080565 47025319389696 controller.py:291] eval | step: 620558 | eval time: 47.3 sec | output: {'test_accuracy': 0.68068, 'test_loss': 0.3297141}
I0719 02:41:31.931695 47025319389696 controller.py:291] eval | step: 630567 | eval time: 47.5 sec | output: {'test_accuracy': 0.67544, 'test_loss': 0.33620217}
I0719 03:18:53.615640 47025319389696 controller.py:291] eval | step: 640576 | eval time: 47.8 sec | output: {'test_accuracy': 0.67902, 'test_loss': 0.32960042}
I0719 03:56:03.865268 47025319389696 controller.py:291] eval | step: 650585 | eval time: 47.2 sec | output: {'test_accuracy': 0.68312, 'test_loss': 0.32697994}
I0719 04:33:13.600398 47025319389696 controller.py:291] eval | step: 660594 | eval time: 49.1 sec | output: {'test_accuracy': 0.68692, 'test_loss': 0.3218242}
I0719 05:10:26.761108 47025319389696 controller.py:291] eval | step: 670603 | eval time: 47.5 sec | output: {'test_accuracy': 0.68098, 'test_loss': 0.33020604}
I0719 05:47:24.072698 47025319389696 controller.py:291] eval | step: 680612 | eval time: 47.8 sec | output: {'test_accuracy': 0.68784, 'test_loss': 0.32333767}
I0719 06:24:23.891359 47025319389696 controller.py:291] eval | step: 690621 | eval time: 47.1 sec | output: {'test_accuracy': 0.68598, 'test_loss': 0.32332215}
I0719 07:01:34.794637 47025319389696 controller.py:291] eval | step: 700630 | eval time: 48.9 sec | output: {'test_accuracy': 0.68566, 'test_loss': 0.3250208}
I0719 07:38:47.115363 47025319389696 controller.py:291] eval | step: 710639 | eval time: 47.2 sec | output: {'test_accuracy': 0.68696, 'test_loss': 0.32350788}
I0719 08:15:57.504423 47025319389696 controller.py:291] eval | step: 720648 | eval time: 52.0 sec | output: {'test_accuracy': 0.68832, 'test_loss': 0.3217886}
I0719 08:53:14.727922 47025319389696 controller.py:291] eval | step: 730657 | eval time: 49.9 sec | output: {'test_accuracy': 0.6852, 'test_loss': 0.32565758}
I0719 09:30:20.804430 47025319389696 controller.py:291] eval | step: 740666 | eval time: 49.2 sec | output: {'test_accuracy': 0.68326, 'test_loss': 0.33065847}
I0719 10:07:20.642869 47025319389696 controller.py:291] eval | step: 750675 | eval time: 48.0 sec | output: {'test_accuracy': 0.68372, 'test_loss': 0.32719776}
I0719 10:46:23.875730 47025319389696 controller.py:291] eval | step: 760684 | eval time: 47.9 sec | output: {'test_accuracy': 0.68192, 'test_loss': 0.32962936}
I0719 11:23:17.199512 47025319389696 controller.py:291] eval | step: 770693 | eval time: 47.2 sec | output: {'test_accuracy': 0.68262, 'test_loss': 0.32934195}
I0719 12:00:04.367842 47025319389696 controller.py:291] eval | step: 780702 | eval time: 46.9 sec | output: {'test_accuracy': 0.67706, 'test_loss': 0.3367395}
I0719 12:37:04.218576 47025319389696 controller.py:291] eval | step: 790711 | eval time: 49.9 sec | output: {'test_accuracy': 0.68592, 'test_loss': 0.32515442}
I0719 13:14:02.169136 47025319389696 controller.py:291] eval | step: 800720 | eval time: 49.5 sec | output: {'test_accuracy': 0.6774, 'test_loss': 0.33403492}
I0719 13:51:13.950588 47025319389696 controller.py:291] eval | step: 810729 | eval time: 48.3 sec | output: {'test_accuracy': 0.68796, 'test_loss': 0.32154444}
I0719 14:28:19.452734 47025319389696 controller.py:291] eval | step: 820738 | eval time: 47.6 sec | output: {'test_accuracy': 0.68378, 'test_loss': 0.32751894}
I0719 15:05:49.973363 47025319389696 controller.py:291] eval | step: 830747 | eval time: 47.5 sec | output: {'test_accuracy': 0.6856, 'test_loss': 0.3268724}
I0719 15:42:51.213348 47025319389696 controller.py:291] eval | step: 840756 | eval time: 46.4 sec | output: {'test_accuracy': 0.68698, 'test_loss': 0.32407942}
I0719 16:19:54.159197 47025319389696 controller.py:291] eval | step: 850765 | eval time: 48.8 sec | output: {'test_accuracy': 0.6875, 'test_loss': 0.32387343}
I0719 16:57:06.514332 47025319389696 controller.py:291] eval | step: 860774 | eval time: 48.9 sec | output: {'test_accuracy': 0.6877, 'test_loss': 0.32420784}
I0719 17:34:12.212647 47025319389696 controller.py:291] eval | step: 870783 | eval time: 48.5 sec | output: {'test_accuracy': 0.688, 'test_loss': 0.3242049}
I0719 18:11:14.854687 47025319389696 controller.py:291] eval | step: 880792 | eval time: 48.6 sec | output: {'test_accuracy': 0.6882, 'test_loss': 0.324394}
I0719 18:48:31.401537 47025319389696 controller.py:291] eval | step: 890801 | eval time: 49.7 sec | output: {'test_accuracy': 0.68516, 'test_loss': 0.32722312}
I0719 19:25:37.033689 47025319389696 controller.py:291] eval | step: 900810 | eval time: 48.7 sec | output: {'test_accuracy': 0.68694, 'test_loss': 0.32629344}
eval | step: 10009 | eval time: 74.7 sec | output: {'test_accuracy': 0.08882, 'test_loss': 1.2578498}
eval | step: 20018 | eval time: 47.9 sec | output: {'test_accuracy': 0.18752, 'test_loss': 1.0495228}
eval | step: 30027 | eval time: 48.0 sec | output: {'test_accuracy': 0.2584, 'test_loss': 0.9102226}
eval | step: 40036 | eval time: 47.3 sec | output: {'test_accuracy': 0.33168, 'test_loss': 0.77455497}
eval | step: 50045 | eval time: 48.6 sec | output: {'test_accuracy': 0.32974, 'test_loss': 0.7866249}
eval | step: 60054 | eval time: 48.0 sec | output: {'test_accuracy': 0.32832, 'test_loss': 0.7928696}
eval | step: 70063 | eval time: 47.4 sec | output: {'test_accuracy': 0.36952, 'test_loss': 0.7357325}
eval | step: 80072 | eval time: 49.6 sec | output: {'test_accuracy': 0.33716, 'test_loss': 0.7839137}
eval | step: 90081 | eval time: 51.2 sec | output: {'test_accuracy': 0.32314, 'test_loss': 0.82485163}
eval | step: 100090 | eval time: 49.1 sec | output: {'test_accuracy': 0.36102, 'test_loss': 0.7599265}
eval | step: 110099 | eval time: 49.5 sec | output: {'test_accuracy': 0.42732, 'test_loss': 0.6477533}
eval | step: 120108 | eval time: 49.1 sec | output: {'test_accuracy': 0.3769, 'test_loss': 0.745124}
eval | step: 130117 | eval time: 196.2 sec | output: {'test_accuracy': 0.43222, 'test_loss': 0.63742775}
eval | step: 140126 | eval time: 49.1 sec | output: {'test_accuracy': 0.43536, 'test_loss': 0.63410264}
eval | step: 150135 | eval time: 46.7 sec | output: {'test_accuracy': 0.43782, 'test_loss': 0.6309778}
eval | step: 160144 | eval time: 49.5 sec | output: {'test_accuracy': 0.44036, 'test_loss': 0.6212317}
eval | step: 170153 | eval time: 47.7 sec | output: {'test_accuracy': 0.4211, 'test_loss': 0.6638591}
eval | step: 180162 | eval time: 48.7 sec | output: {'test_accuracy': 0.41044, 'test_loss': 0.6806779}
eval | step: 190171 | eval time: 48.9 sec | output: {'test_accuracy': 0.4227, 'test_loss': 0.6592407}
eval | step: 200180 | eval time: 47.1 sec | output: {'test_accuracy': 0.41614, 'test_loss': 0.6689654}
eval | step: 210189 | eval time: 49.0 sec | output: {'test_accuracy': 0.43086, 'test_loss': 0.6471556}
eval | step: 220198 | eval time: 49.0 sec | output: {'test_accuracy': 0.43288, 'test_loss': 0.651372}
eval | step: 230207 | eval time: 48.6 sec | output: {'test_accuracy': 0.44876, 'test_loss': 0.61514467}
eval | step: 240216 | eval time: 47.8 sec | output: {'test_accuracy': 0.39994, 'test_loss': 0.6896438}
eval | step: 250225 | eval time: 49.3 sec | output: {'test_accuracy': 0.43974, 'test_loss': 0.63887787}
eval | step: 260234 | eval time: 48.7 sec | output: {'test_accuracy': 0.42148, 'test_loss': 0.65387714}
eval | step: 270243 | eval time: 48.0 sec | output: {'test_accuracy': 0.42988, 'test_loss': 0.64366835}
eval | step: 280252 | eval time: 49.1 sec | output: {'test_accuracy': 0.4292, 'test_loss': 0.64544094}
eval | step: 290261 | eval time: 47.7 sec | output: {'test_accuracy': 0.43688, 'test_loss': 0.633886}
eval | step: 300270 | eval time: 47.0 sec | output: {'test_accuracy': 0.4378, 'test_loss': 0.6427796}
eval | step: 310279 | eval time: 47.1 sec | output: {'test_accuracy': 0.60604, 'test_loss': 0.41581097}
eval | step: 320288 | eval time: 47.6 sec | output: {'test_accuracy': 0.60958, 'test_loss': 0.41247183}
eval | step: 330297 | eval time: 47.5 sec | output: {'test_accuracy': 0.62418, 'test_loss': 0.3925642}
eval | step: 340306 | eval time: 49.3 sec | output: {'test_accuracy': 0.60878, 'test_loss': 0.41140762}
eval | step: 350315 | eval time: 46.3 sec | output: {'test_accuracy': 0.61416, 'test_loss': 0.40241298}
eval | step: 360324 | eval time: 48.7 sec | output: {'test_accuracy': 0.61666, 'test_loss': 0.40103617}
eval | step: 370333 | eval time: 47.6 sec | output: {'test_accuracy': 0.64238, 'test_loss': 0.3717267}
eval | step: 380342 | eval time: 46.9 sec | output: {'test_accuracy': 0.63504, 'test_loss': 0.37905875}
eval | step: 390351 | eval time: 46.7 sec | output: {'test_accuracy': 0.62588, 'test_loss': 0.38983408}
eval | step: 400360 | eval time: 47.5 sec | output: {'test_accuracy': 0.62752, 'test_loss': 0.39042756}
eval | step: 410369 | eval time: 47.6 sec | output: {'test_accuracy': 0.63086, 'test_loss': 0.38619643}
eval | step: 420378 | eval time: 46.4 sec | output: {'test_accuracy': 0.61492, 'test_loss': 0.40776753}
eval | step: 430387 | eval time: 47.1 sec | output: {'test_accuracy': 0.61482, 'test_loss': 0.40698445}
eval | step: 440396 | eval time: 46.6 sec | output: {'test_accuracy': 0.61064, 'test_loss': 0.40885204}
eval | step: 450405 | eval time: 47.2 sec | output: {'test_accuracy': 0.6101, 'test_loss': 0.41232613}
eval | step: 460414 | eval time: 50.2 sec | output: {'test_accuracy': 0.62022, 'test_loss': 0.39717302}
eval | step: 470423 | eval time: 47.8 sec | output: {'test_accuracy': 0.63738, 'test_loss': 0.3755849}
eval | step: 480432 | eval time: 48.6 sec | output: {'test_accuracy': 0.63156, 'test_loss': 0.38420147}
eval | step: 490441 | eval time: 47.9 sec | output: {'test_accuracy': 0.61762, 'test_loss': 0.40265194}
eval | step: 500450 | eval time: 47.4 sec | output: {'test_accuracy': 0.63422, 'test_loss': 0.3809521}
eval | step: 510459 | eval time: 50.4 sec | output: {'test_accuracy': 0.62098, 'test_loss': 0.39487627}
eval | step: 520468 | eval time: 47.2 sec | output: {'test_accuracy': 0.61512, 'test_loss': 0.40484685}
eval | step: 530477 | eval time: 47.3 sec | output: {'test_accuracy': 0.63776, 'test_loss': 0.37440324}
eval | step: 540486 | eval time: 203.9 sec | output: {'test_accuracy': 0.62722, 'test_loss': 0.39110208}
eval | step: 550495 | eval time: 47.9 sec | output: {'test_accuracy': 0.63562, 'test_loss': 0.38335004}
eval | step: 560504 | eval time: 46.5 sec | output: {'test_accuracy': 0.61826, 'test_loss': 0.40100682}
eval | step: 570513 | eval time: 46.1 sec | output: {'test_accuracy': 0.63496, 'test_loss': 0.3817627}
eval | step: 580522 | eval time: 47.2 sec | output: {'test_accuracy': 0.61354, 'test_loss': 0.40860355}
eval | step: 590531 | eval time: 48.7 sec | output: {'test_accuracy': 0.62766, 'test_loss': 0.3902482}
eval | step: 600540 | eval time: 47.3 sec | output: {'test_accuracy': 0.62992, 'test_loss': 0.38722458}
eval | step: 610549 | eval time: 49.5 sec | output: {'test_accuracy': 0.6785, 'test_loss': 0.3319877}
eval | step: 620558 | eval time: 47.3 sec | output: {'test_accuracy': 0.68068, 'test_loss': 0.3297141}
eval | step: 630567 | eval time: 47.5 sec | output: {'test_accuracy': 0.67544, 'test_loss': 0.33620217}
eval | step: 640576 | eval time: 47.8 sec | output: {'test_accuracy': 0.67902, 'test_loss': 0.32960042}
eval | step: 650585 | eval time: 47.2 sec | output: {'test_accuracy': 0.68312, 'test_loss': 0.32697994}
eval | step: 660594 | eval time: 49.1 sec | output: {'test_accuracy': 0.68692, 'test_loss': 0.3218242}
eval | step: 670603 | eval time: 47.5 sec | output: {'test_accuracy': 0.68098, 'test_loss': 0.33020604}
eval | step: 680612 | eval time: 47.8 sec | output: {'test_accuracy': 0.68784, 'test_loss': 0.32333767}
eval | step: 690621 | eval time: 47.1 sec | output: {'test_accuracy': 0.68598, 'test_loss': 0.32332215}
eval | step: 700630 | eval time: 48.9 sec | output: {'test_accuracy': 0.68566, 'test_loss': 0.3250208}
eval | step: 710639 | eval time: 47.2 sec | output: {'test_accuracy': 0.68696, 'test_loss': 0.32350788}
eval | step: 720648 | eval time: 52.0 sec | output: {'test_accuracy': 0.68832, 'test_loss': 0.3217886}
eval | step: 730657 | eval time: 49.9 sec | output: {'test_accuracy': 0.6852, 'test_loss': 0.32565758}
eval | step: 740666 | eval time: 49.2 sec | output: {'test_accuracy': 0.68326, 'test_loss': 0.33065847}
eval | step: 750675 | eval time: 48.0 sec | output: {'test_accuracy': 0.68372, 'test_loss': 0.32719776}
eval | step: 760684 | eval time: 47.9 sec | output: {'test_accuracy': 0.68192, 'test_loss': 0.32962936}
eval | step: 770693 | eval time: 47.2 sec | output: {'test_accuracy': 0.68262, 'test_loss': 0.32934195}
eval | step: 780702 | eval time: 46.9 sec | output: {'test_accuracy': 0.67706, 'test_loss': 0.3367395}
eval | step: 790711 | eval time: 49.9 sec | output: {'test_accuracy': 0.68592, 'test_loss': 0.32515442}
eval | step: 800720 | eval time: 49.5 sec | output: {'test_accuracy': 0.6774, 'test_loss': 0.33403492}
eval | step: 810729 | eval time: 48.3 sec | output: {'test_accuracy': 0.68796, 'test_loss': 0.32154444}
eval | step: 820738 | eval time: 47.6 sec | output: {'test_accuracy': 0.68378, 'test_loss': 0.32751894}
eval | step: 830747 | eval time: 47.5 sec | output: {'test_accuracy': 0.6856, 'test_loss': 0.3268724}
eval | step: 840756 | eval time: 46.4 sec | output: {'test_accuracy': 0.68698, 'test_loss': 0.32407942}
eval | step: 850765 | eval time: 48.8 sec | output: {'test_accuracy': 0.6875, 'test_loss': 0.32387343}
eval | step: 860774 | eval time: 48.9 sec | output: {'test_accuracy': 0.6877, 'test_loss': 0.32420784}
eval | step: 870783 | eval time: 48.5 sec | output: {'test_accuracy': 0.688, 'test_loss': 0.3242049}
eval | step: 880792 | eval time: 48.6 sec | output: {'test_accuracy': 0.6882, 'test_loss': 0.324394}
eval | step: 890801 | eval time: 49.7 sec | output: {'test_accuracy': 0.68516, 'test_loss': 0.32722312}
eval | step: 900810 | eval time: 48.7 sec | output: {'test_accuracy': 0.68694, 'test_loss': 0.32629344}