tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [predictions must be <= 1] [Condition x <= y did not hold element-wise:] [x (dssm/dot/Squeeze:0) = ] [[0][0][0.291515917]...] [y (Cast_6/x:0) = ] [1]

It fits normally in first several epochs, but in last epoch, it runs error:

Epoch 37/40
389/389 [==============================] - 1s 2ms/step - loss: 0.3662 - auc: 0.9522
Epoch 38/40
389/389 [==============================] - 1s 2ms/step - loss: 0.3769 - auc: 0.9490
Epoch 39/40
389/389 [==============================] - 1s 2ms/step - loss: 0.3695 - auc: 0.9525
Epoch 40/40
 29/389 [=>............................] - ETA: 0s - loss: 0.3429 - auc: 0.9625Traceback (most recent call last):
  File "/mnt1/liulei/mytest/dssm_stream.py", line 412, in 
    train_and_test()
  File "/mnt1/liulei/mytest/dssm_stream.py", line 365, in train_and_test
    dssm.fit([user_features_train_x, item_features_train_x], train_y, epochs=40, batch_size=64, class_weight={0: 1, 1: 2})
  File "/mnt/softwares/tf2/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 108, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/mnt/softwares/tf2/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1098, in fit
    tmp_logs = train_function(iterator)
  File "/mnt/softwares/tf2/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "/mnt/softwares/tf2/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 807, in _call
    return self._stateless_fn(*args, **kwds)  # pylint: disable=not-callable
  File "/mnt/softwares/tf2/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2829, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/mnt/softwares/tf2/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1848, in _filtered_call
    cancellation_manager=cancellation_manager)
  File "/mnt/softwares/tf2/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1924, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/mnt/softwares/tf2/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 550, in call
    ctx=ctx)
  File "/mnt/softwares/tf2/lib/python3.6/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError:  assertion failed: [predictions must be <= 1] [Condition x <= y did not hold element-wise:] [x (dssm/dot/Squeeze:0) = ] [[0.47509262][0][0]...] [y (Cast_6/x:0) = ] [1]
	 [[{{node assert_less_equal/Assert/AssertGuard/else/_11/assert_less_equal/Assert/AssertGuard/Assert}}]] [Op:__inference_train_function_2753]

Function call stack:
train_function

22/06/01 16:59:33 INFO ShutdownHookManager: Shutdown hook called
22/06/01 16:59:33 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-2d895225-7f09-4357-b1c3-8e7fcceb2816/pyspark-08519124-2e92-4c74-ae57-43f18a9234d4
22/06/01 16:59:33 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-2d895225-7f09-4357-b1c3-8e7fcceb2816
22/06/01 16:59:33 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-885bb7d9-5dfe-4f7b-a261-fb5714e87713

model shape:

__________________________________________________________________________________________________
Model: "dssm"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
user_input (InputLayer)         [(None, 67)]         0                                            
__________________________________________________________________________________________________
item_input (InputLayer)         [(None, 32)]         0                                            
__________________________________________________________________________________________________
tf_op_layer_strided_slice_2 (Te [(None, 36)]         0           user_input[0][0]                 
__________________________________________________________________________________________________
tf_op_layer_strided_slice_6 (Te [(None, 7)]          0           item_input[0][0]                 
__________________________________________________________________________________________________
tf_op_layer_strided_slice (Tens [(None,)]            0           user_input[0][0]                 
__________________________________________________________________________________________________
se_net_block (SENetBlock)       (None, 36)           488         tf_op_layer_strided_slice_2[0][0]
__________________________________________________________________________________________________
tf_op_layer_strided_slice_3 (Te [(None, 36)]         0           user_input[0][0]                 
__________________________________________________________________________________________________
tf_op_layer_strided_slice_4 (Te [(None,)]            0           item_input[0][0]                 
__________________________________________________________________________________________________
se_net_block_1 (SENetBlock)     (None, 7)            83          tf_op_layer_strided_slice_6[0][0]
__________________________________________________________________________________________________
tf_op_layer_strided_slice_7 (Te [(None, 7)]          0           item_input[0][0]                 
__________________________________________________________________________________________________
embedding (Embedding)           (None, 32)           105088      tf_op_layer_strided_slice[0][0]  
__________________________________________________________________________________________________
tf_op_layer_strided_slice_1 (Te [(None, 30)]         0           user_input[0][0]                 
__________________________________________________________________________________________________
tf_op_layer_Mul (TensorFlowOpLa [(None, 36)]         0           se_net_block[0][0]               
                                                                 tf_op_layer_strided_slice_3[0][0]
__________________________________________________________________________________________________
embedding_2 (Embedding)         (None, 32)           27456       tf_op_layer_strided_slice_4[0][0]
__________________________________________________________________________________________________
tf_op_layer_strided_slice_5 (Te [(None, 24)]         0           item_input[0][0]                 
__________________________________________________________________________________________________
tf_op_layer_Mul_1 (TensorFlowOp [(None, 7)]          0           se_net_block_1[0][0]             
                                                                 tf_op_layer_strided_slice_7[0][0]
__________________________________________________________________________________________________
concatenate (Concatenate)       (None, 98)           0           embedding[0][0]                  
                                                                 tf_op_layer_strided_slice_1[0][0]
                                                                 tf_op_layer_Mul[0][0]            
__________________________________________________________________________________________________
concatenate_1 (Concatenate)     (None, 63)           0           embedding_2[0][0]                
                                                                 tf_op_layer_strided_slice_5[0][0]
                                                                 tf_op_layer_Mul_1[0][0]          
__________________________________________________________________________________________________
dense_2 (Dense)                 (None, 32)           3168        concatenate[0][0]                
__________________________________________________________________________________________________
dense_6 (Dense)                 (None, 32)           2048        concatenate_1[0][0]              
__________________________________________________________________________________________________
dense_3 (Dense)                 (None, 32)           1056        dense_2[0][0]                    
__________________________________________________________________________________________________
dense_7 (Dense)                 (None, 32)           1056        dense_6[0][0]                    
__________________________________________________________________________________________________
dot (Dot)                       (None, 1)            0           dense_3[0][0]                    
                                                                 dense_7[0][0]                    
==================================================================================================

I trained DSSM model, and output the cosine similarity :

    ...
cos_similarity = tf.keras.layers.Dot(axes=(1, 1), normalize=True)([user_output, item_output])
dssm = Model([user_input, item_input], cos_similarity, name="dssm")
return user_model, item_model, dssm