Latent_Dirichlet_Allocation model doesn't train

Hello,

I try to run the tutorial example for LDA with the default parameters, I have the same issue as in #110 but my packages are up to date (tf 1.14.0 , tfp 0.7.0-rc0).

Here are the logs when I run the code :
`

, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x000002F3D4020940>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
W0619 17:54:47.889011 23232 deprecation.py:323] From C:\Users\JohnSmith\Anaconda3\lib\site-packages\tensorflow\python\training\training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0619 17:54:48.328576 23232 deprecation.py:323] From C:/Users/JohnSmith/Documents/mail/email_classification/scripts/TFPLDA.py:432: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means tf.py_functions can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.

I0619 17:54:48.350521 23232 estimator.py:1145] Calling model_fn.
W0619 17:54:48.354504 23232 deprecation.py:506] From C:\Users\JohnSmith\Anaconda3\lib\site-packages\tensorflow\python\ops\init_ops.py:1288: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0619 17:54:48.584355 23232 deprecation.py:323] From C:\Users\JohnSmith\Anaconda3\lib\site-packages\tensorflow\python\ops\math_grad.py:1250: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
I0619 17:54:48.859852 23232 estimator.py:1147] Done calling model_fn.
I0619 17:54:48.859852 23232 basic_session_run_hooks.py:541] Create CheckpointSaverHook.
I0619 17:54:49.026406 23232 monitored_session.py:240] Graph was finalized.
2019-06-19 17:54:49.027551: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
W0619 17:54:49.032390 23232 deprecation.py:323] From C:\Users\JohnSmith\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
I0619 17:54:49.036174 23232 saver.py:1280] Restoring parameters from ../_tmp\lda/model.ckpt-60000
W0619 17:54:49.092713 23232 deprecation.py:323] From C:\Users\JohnSmith\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py:1066: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file utilities to get mtimes.
I0619 17:54:49.128959 23232 session_manager.py:500] Running local_init_op.
I0619 17:54:49.145915 23232 session_manager.py:502] Done running local_init_op.
I0619 17:54:49.481188 23232 basic_session_run_hooks.py:606] Saving checkpoints for 60000 into ../_tmp\lda/model.ckpt.
I0619 17:54:49.812525 23232 basic_session_run_hooks.py:262] loss = 708.6334, step = 60001
I0619 17:54:50.635037 23232 basic_session_run_hooks.py:692] global_step/sec: 121.579
I0619 17:54:50.636033 23232 basic_session_run_hooks.py:260] loss = 869.03516, step = 60101 (0.824 sec)
I0619 17:54:51.374059 23232 basic_session_run_hooks.py:692] global_step/sec: 135.314
I0619 17:54:51.374059 23232 basic_session_run_hooks.py:260] loss = 469.73938, step = 60201 (0.738 sec)
I0619 17:54:52.110108 23232 basic_session_run_hooks.py:692] global_step/sec: 136.044
I0619 17:54:52.110108 23232 basic_session_run_hooks.py:260] loss = 497.04263, step = 60301 (0.736 sec)
I0619 17:54:52.855114 23232 basic_session_run_hooks.py:692] global_step/sec: 134.049
I0619 17:54:52.855114 23232 basic_session_run_hooks.py:260] loss = 638.5796, step = 60401 (0.745 sec)
I0619 17:54:53.786989 23232 basic_session_run_hooks.py:692] global_step/sec: 107.311
I0619 17:54:53.787947 23232 basic_session_run_hooks.py:260] loss = 565.28516, step = 60501 (0.933 sec)
I0619 17:54:54.865794 23232 basic_session_run_hooks.py:692] global_step/sec: 92.7811
I0619 17:54:54.866791 23232 basic_session_run_hooks.py:260] loss = 593.31055, step = 60601 (1.079 sec)
I0619 17:54:55.911526 23232 basic_session_run_hooks.py:692] global_step/sec: 95.5356
I0619 17:54:55.912522 23232 basic_session_run_hooks.py:260] loss = 889.1388, step = 60701 (1.046 sec)
I0619 17:54:56.960004 23232 basic_session_run_hooks.py:692] global_step/sec: 95.3763
I0619 17:54:56.961001 23232 basic_session_run_hooks.py:260] loss = 788.23865, step = 60801 (1.048 sec)
I0619 17:54:58.012189 23232 basic_session_run_hooks.py:692] global_step/sec: 95.0403
I0619 17:54:58.013187 23232 basic_session_run_hooks.py:260] loss = 476.56763, step = 60901 (1.052 sec)
I0619 17:54:59.085863 23232 basic_session_run_hooks.py:692] global_step/sec: 93.1381
I0619 17:54:59.086860 23232 basic_session_run_hooks.py:260] loss = 434.7486, step = 61001 (1.074 sec)
I0619 17:55:00.216380 23232 basic_session_run_hooks.py:692] global_step/sec: 88.4551
I0619 17:55:00.217375 23232 basic_session_run_hooks.py:260] loss = 509.8182, step = 61101
...............................
(1.133 sec)
I0619 17:56:44.382706 23232 basic_session_run_hooks.py:692] global_step/sec: 88.6349
I0619 17:56:44.383702 23232 basic_session_run_hooks.py:260] loss = 469.25922, step = 69701 (1.128 sec)
I0619 17:56:45.521680 23232 basic_session_run_hooks.py:692] global_step/sec: 87.7983
I0619 17:56:45.522679 23232 basic_session_run_hooks.py:260] loss = 515.9735, step = 69801 (1.139 sec)
I0619 17:56:46.661003 23232 basic_session_run_hooks.py:692] global_step/sec: 87.7714
I0619 17:56:46.662000 23232 basic_session_run_hooks.py:260] loss = 425.97827, step = 69901 (1.139 sec)
I0619 17:56:47.775023 23232 basic_session_run_hooks.py:606] Saving checkpoints for 70000 into ../_tmp\lda/model.ckpt.
W0619 17:56:47.827904 23232 deprecation.py:323] From C:\Users\JohnSmith\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py:960: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to delete files with this prefix.
I0619 17:56:47.959602 23232 estimator.py:368] Loss for final step: 637.4798.
I0619 17:56:48.403129 23232 estimator.py:1145] Calling model_fn.
I0619 17:56:49.078269 23232 estimator.py:1147] Done calling model_fn.
I0619 17:56:49.095222 23232 evaluation.py:255] Starting evaluation at 2019-06-19T17:56:49Z
I0619 17:56:49.205926 23232 monitored_session.py:240] Graph was finalized.
I0619 17:56:49.208929 23232 saver.py:1280] Restoring parameters from ../_tmp\lda/model.ckpt-70000
I0619 17:56:49.300813 23232 session_manager.py:500] Running local_init_op.
I0619 17:56:49.321756 23232 session_manager.py:502] Done running local_init_op.
I0619 17:56:51.481999 23232 evaluation.py:275] Finished evaluation at 2019-06-19-17:56:51
I0619 17:56:51.481999 23232 estimator.py:2039] Saving dict for global step 70000: elbo = -581.07947, global_step = 70000, kl = 4.539948, log_likelihood = -576.5394, loss = 581.13477, perplexity = 1021.374, topics = [b'index=0 alpha=0.70 write one get article know use like make say think'
b'index=1 alpha=0.70 write get one article know like use make think go\x00'
b'index=2 alpha=0.70 write get article one like know use go think make\x00'
b'index=3 alpha=0.70 write one get article like use know think make go\x00'
b'index=4 alpha=0.70 write one article get know think use like make say'
b'index=5 alpha=0.70 one write get article use time like call say know\x00'
b'index=6 alpha=0.70 write one get article like know use think make say'
b'index=7 alpha=0.70 write one article get know like use think make go\x00'
b'index=8 alpha=0.70 write article get one like know use think make go\x00'
b'index=9 alpha=0.70 write one article get know like use make think say']
elbo
-581.07947

kl
4.539948

log_likelihood
-576.5394

loss
581.13477

perplexity
1021.374

topics
b'index=0 alpha=0.70 write one get article know use like make say think'
b'index=1 alpha=0.70 write get one article know like use make think go\x00'
b'index=2 alpha=0.70 write get article one like know use go think make\x00'
b'index=3 alpha=0.70 write one get article like use know think make go\x00'
b'index=4 alpha=0.70 write one article get know think use like make say'
b'index=5 alpha=0.70 one write get article use time like call say know\x00'
b'index=6 alpha=0.70 write one get article like know use think make say'
b'index=7 alpha=0.70 write one article get know like use think make go\x00'
b'index=8 alpha=0.70 write article get one like know use think make go\x00'
b'index=9 alpha=0.70 write one article get know like use make think say'

global_step
70000
`

I am still a newbie with TensorFlow

Thanks for your help

Start with the tutorial example and the referenced #110 issue; inspect scripts/TFPLDA.py around line 432, which appears in the log. Reproduce with TensorFlow 1.14.0 and TFP 0.7.0-rc0, then verify that the LDA model trains as expected rather than producing indistinguishable topics.