遇到线上tdm serving evaluation结果和线下模型predict评测结果差异巨大

**目前使用真实生产数据，tdm att的线上和线下(分布式)流程都跑通了(cpu版本)**
线下部分使用的是:x-deeplearning/xdl-algorithm-solution/TDM/script/tdm_ub_att_ubuntu
线上部分模型转换使用的是:x-deeplearning/blaze/tools/example_model/tdm_att
线上评估使用的是:x-deeplearning/xdl-algorithm-solution/TDMServing/evaluation

**线下离线模型predict结果为:**
```
predict result:
global_sample_num: 2548
global_r_num: 3794
global_gt_num: 40368
global_p_num: 254800
global_r: 0.093985
global_p: 0.014890
avg_r: 0.115653
avg_p: 0.014890
```

**上周用tdm serving evaluation跑同样的测试集结果基本不可用**
```
avg_precision: 0.00141051
avg_recall: 0.0228302
avg_f1_score: 0.00256338
```

查看线上和线下树检索的配置参数，也是对齐的，树每层取400,最终取top200个进行评估。

因为在模型转换(model_converter_example.sh)的时候，遇见(**error:name is not contained in mxnet data=fc_b_1** )错误，发现是fc_b_1和fc_b_2没有成功存到线下模型产出的dense文件中，当时做法是直接把graph_ulf.txt对应的几项bias删除掉，能够正常转换

后来回想，以为是此问题导致的线上和线下指标对不齐。

随后追查发现离线模型导出参数，fc_b_1和fc_b_2并没有成功保存到checkpoint中，然后对tdm_ub_att_ubuntu中train.py相关进行了修改，能够成功保存fc_b_1和fc_b_2。
代码改动如下.
tdm_layer_master.py FullyConnected3D中的代码进行了改动，改动如下：

`self.bias = mx.sym.ones(shape=(1, self.output_dim)) * 0.1`
---->
`self.bias = mx.sym.var(name='fc_b_%s' % self.version, shape=(1, self.output_dim), init=mx.init.Constant(0.1))`

重新训练，最终线下模型predict结果和修改前基本无差异，紧接着模型转换也不会出现变量找不到的错误，但是线上评估结果依旧很差。

**另外根据代码可知，线下predict评测使用的是最后一次checkpoint数据，而线上使用的是worker 0导出的模型文件**

**目前发现线上预估打分最终top item的score都是1。**

**同一条测试样本，线下top item打分如下:**

`2949127_T:20463, 0.999499;50142, 0.999482;10959, 0.999434;83256, 0.999244;16435, 0.999200;30899, 0.999035;101882, 0.998858;59799, 0.998675;12190, 0.998552;40287, 0.998492;19108, 0.998490;78018, 0.998479;2 0540, 0.998329;95618, 0.998156;64900, 0.998137;37469, 0.998074;38073, 0.998044;42087, 0.997949;30902, 0.997931;54074, 0.997561;20843, 0.997425;59043, 0.997383;45780, 0.997378;23563, 0.997245;89596, 0.9971 24;20541, 0.997077;19778, 0.997012;20465, 0.996992;46399, 0.996713;27555, 0.996561;29483, 0.996416;32650, 0.996198;25304, 0.996149;56910, 0.996009;85521, 0.995945;12093, 0.995805;48153, 0.995743;54072, 0.995560;83236, 0.995437;76131, 0.995380;31814, 0.995288;27888, 0.995282;83262, 0.995143;20504, 0.994918;43837, 0.994917;61074, 0.994676;28350, 0.994396;39025, 0.994335;20473, 0.994318;41505, 0.993801;36632, 0.993600;96427, 0.993572;83223, 0.993534;15597, 0.993285;77203, 0.993166;61453, 0.992992;15603, 0.992714;33379, 0.992648;75105, 0.991894;17636, 0.991852;40495, 0.991694;17307, 0.991462;52800, 0.991134;52674, 0.990924;49948, 0.990409;76146, 0.990066;74603, 0.989750;13273, 0.989550;17716, 0.989269;25253, 0.988323;29477, 0.987356;16444, 0.985858;76168, 0.985582;42078, 0.985433;62847, 0.985287;23562, 0.9842 19;101883, 0.983762;15600, 0.983568;42089, 0.983296;38671, 0.982821;49946, 0.981332;30764, 0.981068;59793, 0.980869;46385, 0.980765;52840, 0.979820;103712, 0.979248;33037, 0.979136;27609, 0.978070;32640, 0.976367;113510, 0.972216;77519, 0.966582;62773, 0.965298;33772, 0.961509;84408, 0.958087;32643, 0.953578;26821, 0.925344;16467, 0.919517;122714, 0.917185;130521, 0.896764;27470, 0.682379;`

**线上top item打分如下：**

```
I1125 16:10:15.730403 20904 tdm_example.cpp:165] Response: res_code: RC_SUCCESS
result_unit {
id: 22551
score: 1
}
result_unit {
id: 52390
score: 1
}
result_unit {
id: 22248
score: 1
}
result_unit {
id: 96753
score: 1
}
result_unit {
id: 76728
score: 1
}
```
...........

烦请帮忙定位下。

@lovickie @zhuhan1236 @songyue1104 @MaButing

Reproduce the mismatch using TDM/script/tdm_ub_att_ubuntu, TDMServing/evaluation, and the model_converter_example.sh flow. Compare tdm_layer_master.py, the exported checkpoint and worker 0 model, then inspect tdm_example.cpp output against the offline scores; done means identifying and correcting the cause of the differing rankings and scores.

Written by the indexing model from the issue text.