`asset_path_initializer` not fed when saving/reloading model

---- This is an issue posted by @maziyarpanahi ----

Hi @karllessard

We are experiencing the same issue however in the 0.4.0 release (after this fix https://github.com/tensorflow/java/pull/393 was included). Previously in 0.2.x and 0.3.x, we were able to load a model, save it on disk, and load it back (both models from TF Hub and HuggingFace as SavedModel).

However, now in 0.4.0, we can load any SavedModel from anywhere, but once it's saved on disk we face this similar error when trying to load it back (only TF Hub models):

  [ERROR] org.tensorflow.exceptions.TFInvalidArgumentException: You must feed a value for placeholder tensor 'asset_path_initializer' with dtype string  [[node asset_path_initializer]]

• Models coming from TF Hub have an extra asset_path_initializer placeholder, they used to work in the 0.2.x and 0.3.x releases without a need to feed it or deal with it in any way.
• Models coming from HuggingFace don't have an asset_path_initializer placeholder, so we don't have an issue with those

BERT on TF Hub:

BERT on HuggingFace:

Is there anything we need to do since 0.4.x is on TensorFlow 2.7.x, I am not sure if this is a change in newer TF or in tensorflow-java.

PS: We don't use any exporters, we saved the model differently. A very quick and simple example of how we do this:

val model = SavedModelBundle.load(modelPath, "serve")

/* Save the model */
val graph = model.graph()
val session = model.session()

val exportDir = Files.createDirectory(Paths.get("./saved_model")).toAbsolutePath.toString

val variablesFile = Paths.get(exportDir, "variables").toString

session.runner
  .addTarget("StatefulPartitionedCall_1")
  .feed("saver_filename", t.createTensor(variablesFile))
  .run()

val graphFile = Paths.get(exportDir, TensorflowWrapper.SavedModelPB).toString
FileUtils.writeByteArrayToFile(new File(graphFile), graph.toGraphDef.toByteArray)

We have been doing this since TF v1 and it has been compatible with all the TF v2 up until 0.4.0, but now if we save it like this (only models from TF Hub and I am guessing those with assets) we see that error)

Originally posted by @maziyarpanahi in https://github.com/tensorflow/java/issues/387#issuecomment-1066628248

使用 SavedModelBundle.load、graph 和 session，以及 issue 中显示的 saver_filename 目标重现该故障，并比较 TF Hub 和 HuggingFace 模型。检查 graph 和变量如何保存到 SavedModelPB 中，然后重新加载。完成标准是：包含 assets 的 TF Hub 模型可以保存并再次加载，且不会出现 asset_path_initializer 错误。