Exception in thread "main" org.tensorflow.exceptions.TensorFlowException: No gradient defined for op: Concat

Concat doesn't have gradients defined, so it's not possible to train with Concat in a deep net, per some discussion on the mailing list, e.g. https://groups.google.com/a/tensorflow.org/g/jvm/c/TTuT3yzoKWs/m/pTQX1w_XAgAJ

Exception in thread "main" org.tensorflow.exceptions.TensorFlowException: No gradient defined for op: Concat. Please see https://www.tensorflow.org/code/tensorflow/cc/gradients/README.md for instructions on how to add C++ gradients.
        at org.tensorflow.internal.c_api.AbstractTF_Status.throwExceptionIfNotOK(AbstractTF_Status.java:101)
        at org.tensorflow.Graph.addGradients(Graph.java:649)
        at org.tensorflow.Graph.addGradients(Graph.java:267)
        at org.tensorflow.Graph.addGradients(Graph.java:301)
        at org.tensorflow.framework.optimizers.Optimizer.computeGradients(Optimizer.java:113)
        at org.tensorflow.framework.optimizers.Optimizer.minimize(Optimizer.java:94)
        at org.tensorflow.framework.optimizers.Optimizer.minimize(Optimizer.java:90)
        ...

However, because zero padding and add both have gradients defined, my current workaround is to make a "fake concat", which zero pads the two vectors to concat, then adds them together. In Scala, this is:

val padded1 = tf.withName("padded1").pad(some_input_of_512_dimensions,
                                         tf.constant(Array(Array(0,0), Array(1024,0))), tf.constant(0.0f)) 
val padded2 = tf.withName("padded2").pad(some_other_input_of_1024_dimensions,
                                         tf.constant(Array(Array(0,0), Array(0,512))), tf.constant(0.0f))
val fake_concat = tf.withName("fake_concat").math.add(padded1, padded2) // add these two vectors together, effectively concat'ing them.  tf.concat(...) doesn't itself have gradients implemented, so doesn't work during training.

Concat's useful for BatchNorm etc, so I mentioned this in https://github.com/tensorflow/java/issues/135#issuecomment-731706408 and the fake concat pseudocode is at https://groups.google.com/a/tensorflow.org/g/jvm/c/TTuT3yzoKWs/m/pTQX1w_XAgAJ

Models with fake concat train and save fine, but model loading can be problematic (and would be a separate ticket). Hopefully this ticket is useful to track adding gradients to Concat.

Start with Graph.java, especially the addGradients entry points, and read the linked TensorFlow C++ gradients instructions to locate how Concat gradients are registered. Reproduce the reported training failure, then verify that a model using Concat can compute gradients without the exception.