`asset_path_initializer` not fed when saving/reloading model

---- This is an issue posted by @maziyarpanahi ----

Hi @karllessard

We are experiencing the same issue however in the 0.4.0 release (after this fix https://github.com/tensorflow/java/pull/393 was included). Previously in 0.2.x and 0.3.x, we were able to load a model, save it on disk, and load it back (both models from TF Hub and HuggingFace as SavedModel).

However, now in 0.4.0, we can load any SavedModel from anywhere, but once it's saved on disk we face this similar error when trying to load it back (only TF Hub models):

  [ERROR] org.tensorflow.exceptions.TFInvalidArgumentException: You must feed a value for placeholder tensor 'asset_path_initializer' with dtype string  [[node asset_path_initializer]]

• Models coming from TF Hub have an extra asset_path_initializer placeholder, they used to work in the 0.2.x and 0.3.x releases without a need to feed it or deal with it in any way.
• Models coming from HuggingFace don't have an asset_path_initializer placeholder, so we don't have an issue with those

BERT on TF Hub:

BERT on HuggingFace:

Is there anything we need to do since 0.4.x is on TensorFlow 2.7.x, I am not sure if this is a change in newer TF or in tensorflow-java.

PS: We don't use any exporters, we saved the model differently. A very quick and simple example of how we do this:

val model = SavedModelBundle.load(modelPath, "serve")

/* Save the model */
val graph = model.graph()
val session = model.session()

val exportDir = Files.createDirectory(Paths.get("./saved_model")).toAbsolutePath.toString

val variablesFile = Paths.get(exportDir, "variables").toString

session.runner
  .addTarget("StatefulPartitionedCall_1")
  .feed("saver_filename", t.createTensor(variablesFile))
  .run()

val graphFile = Paths.get(exportDir, TensorflowWrapper.SavedModelPB).toString
FileUtils.writeByteArrayToFile(new File(graphFile), graph.toGraphDef.toByteArray)

We have been doing this since TF v1 and it has been compatible with all the TF v2 up until 0.4.0, but now if we save it like this (only models from TF Hub and I am guessing those with assets) we see that error)

Originally posted by @maziyarpanahi in https://github.com/tensorflow/java/issues/387#issuecomment-1066628248

Reproduziere den Fehler mit SavedModelBundle.load, dem graph und der session sowie dem im Issue gezeigten saver_filename-Ziel und vergleiche dabei TF Hub- und HuggingFace-Modelle. Untersuche, wie der graph und die Variablen in SavedModelPB gespeichert und anschließend wieder geladen werden. Als abgeschlossen gilt die Aufgabe, wenn TF Hub-Modelle mit assets gespeichert und erneut geladen werden können, ohne dass der Fehler asset_path_initializer auftritt.