TDM的demo报错，core dump

报错信息如下：
在sess.run([train_op, xdl.get_collection(xdl.UPDATE_OPS)])时报错
`>>>>>>>>>>>> 0 >>>>>>>>>>>
Cuda Stream Create Error Error Code: Aborted (core dumped)
`

配置文件如下：
config.train.json:
`{
"job_name": "xdl_tdm",
"dependent_dirs": "/home/zhang/tdm_work/tdm_ub_att_ubuntu",
"script": "train.py",
"docker_image": "registry.cn-hangzhou.aliyuncs.com/xdl/xdl:ubuntu-gpu-mxnet1.3",
"max_local_failover_times": 3,
"max_failover_times": 3,
"min_finish_worker_rate": 100,
"worker": {
"instance_num": 20,
"cpu_cores": 46,
"gpu_cores": 2,
"memory_m": 100000
},
"ps": {
"instance_num": 16,
"cpu_cores": 16,
"gpu_cores": 0,
"memory_m": 64000
},
"checkpoint": {
"output_dir": "zhangheng/tdm_test/tdm_mock_train/checkpoint"
},
"auto_rebalance": {
"enable": "false"
}
}
`

data/tdm.json:
`
{
"train_rawdata_url": "tdm_test/mock_train.csv",
"test_rawdata_url": "tdm_test/mock_test.csv",
"data_dir": "data",
"tree_filename": "userbehavoir_tree.pb",
"train_sample": "userbehavoir_train_sample.dat",
"test_sample": "userbehavoir_test_sample.dat",
"stat_file": "userbehavoir_stat.dat",
"feature_conf": "userbehavoir_fc.json",
"train_sample_segments": 400,
"seq_len": 10,
"min_seq_len": 6,
"parall": 4,
"train_id_label": "train_unit_id",
"test_id_label": "test_unit_id",
"upload_url": "/tdm_work/tdm_ub_att_ubuntu",
"epocs": 1,
"model_url": "user/tdm_test/embed",
"item_id_file": "leaf.id",
"train_mode": "sync",
"learning_rate": 0.001,
"train_batch_size": 30000,
"train_epochs": 10,
"train_threads": 2,
"save_checkpoint_interval": 10,
"predict_batch_size": 100,
"predict_epochs": 1,
"predict_threads": 1,
"saver_ckpt": "",
"tree_store_config": "size=10000000",
"use_hdfs_tree": 0,
"tree_pb": "data/userbehavoir_tree.pb",
"predict_io_pause_num": 10000,
"tdmop_debug": "true",
"tdmop_layer_counts": "0,1,2,3,4,5,6,7,8,9,1,1,1,1,1,1,1,1,2,2,3,7,2",
"_end_": 0
}
`

用pdb调试时发现是在

我们没有用到hadoop，直接用的local的方式，用gdb调试也没有core生成，看不到任何错误信息
不知道这种涉及到底层c++库的错误会是什么原因

Start by reproducing the failure from train.py with config.train.json and data/tdm.json, focusing on the sess.run call and the reported CUDA Stream Create Error. Use the existing pdb and gdb observations to determine whether a core dump or additional native error can be obtained; done means identifying a reproducible cause and documenting the required configuration or fix.

Written by the indexing model from the issue text.