Exception in thread "main" org.tensorflow.exceptions.TensorFlowException: No gradient defined for op: Concat

Concat doesn't have gradients defined, so it's not possible to train with Concat in a deep net, per some discussion on the mailing list, e.g. https://groups.google.com/a/tensorflow.org/g/jvm/c/TTuT3yzoKWs/m/pTQX1w_XAgAJ

Exception in thread "main" org.tensorflow.exceptions.TensorFlowException: No gradient defined for op: Concat. Please see https://www.tensorflow.org/code/tensorflow/cc/gradients/README.md for instructions on how to add C++ gradients.
        at org.tensorflow.internal.c_api.AbstractTF_Status.throwExceptionIfNotOK(AbstractTF_Status.java:101)
        at org.tensorflow.Graph.addGradients(Graph.java:649)
        at org.tensorflow.Graph.addGradients(Graph.java:267)
        at org.tensorflow.Graph.addGradients(Graph.java:301)
        at org.tensorflow.framework.optimizers.Optimizer.computeGradients(Optimizer.java:113)
        at org.tensorflow.framework.optimizers.Optimizer.minimize(Optimizer.java:94)
        at org.tensorflow.framework.optimizers.Optimizer.minimize(Optimizer.java:90)
        ...

However, because zero padding and add both have gradients defined, my current workaround is to make a "fake concat", which zero pads the two vectors to concat, then adds them together. In Scala, this is:

val padded1 = tf.withName("padded1").pad(some_input_of_512_dimensions,
                                         tf.constant(Array(Array(0,0), Array(1024,0))), tf.constant(0.0f)) 
val padded2 = tf.withName("padded2").pad(some_other_input_of_1024_dimensions,
                                         tf.constant(Array(Array(0,0), Array(0,512))), tf.constant(0.0f))
val fake_concat = tf.withName("fake_concat").math.add(padded1, padded2) // add these two vectors together, effectively concat'ing them.  tf.concat(...) doesn't itself have gradients implemented, so doesn't work during training.

Concat's useful for BatchNorm etc, so I mentioned this in https://github.com/tensorflow/java/issues/135#issuecomment-731706408 and the fake concat pseudocode is at https://groups.google.com/a/tensorflow.org/g/jvm/c/TTuT3yzoKWs/m/pTQX1w_XAgAJ

Models with fake concat train and save fine, but model loading can be problematic (and would be a separate ticket). Hopefully this ticket is useful to track adding gradients to Concat.

Bắt đầu với Graph.java, đặc biệt là các điểm vào của addGradients, và đọc hướng dẫn về gradient C++ của TensorFlow được liên kết để xác định cách các gradient của Concat được đăng ký. Tái hiện lỗi huấn luyện đã được báo cáo, sau đó xác minh rằng một model sử dụng Concat có thể tính gradient mà không gặp exception.