`asset_path_initializer` not fed when saving/reloading model

---- This is an issue posted by @maziyarpanahi ----

Hi @karllessard

We are experiencing the same issue however in the 0.4.0 release (after this fix https://github.com/tensorflow/java/pull/393 was included). Previously in 0.2.x and 0.3.x, we were able to load a model, save it on disk, and load it back (both models from TF Hub and HuggingFace as SavedModel).

However, now in 0.4.0, we can load any SavedModel from anywhere, but once it's saved on disk we face this similar error when trying to load it back (only TF Hub models):

  [ERROR] org.tensorflow.exceptions.TFInvalidArgumentException: You must feed a value for placeholder tensor 'asset_path_initializer' with dtype string  [[node asset_path_initializer]]

• Models coming from TF Hub have an extra asset_path_initializer placeholder, they used to work in the 0.2.x and 0.3.x releases without a need to feed it or deal with it in any way.
• Models coming from HuggingFace don't have an asset_path_initializer placeholder, so we don't have an issue with those

BERT on TF Hub:

BERT on HuggingFace:

Is there anything we need to do since 0.4.x is on TensorFlow 2.7.x, I am not sure if this is a change in newer TF or in tensorflow-java.

PS: We don't use any exporters, we saved the model differently. A very quick and simple example of how we do this:

val model = SavedModelBundle.load(modelPath, "serve")

/* Save the model */
val graph = model.graph()
val session = model.session()

val exportDir = Files.createDirectory(Paths.get("./saved_model")).toAbsolutePath.toString

val variablesFile = Paths.get(exportDir, "variables").toString

session.runner
  .addTarget("StatefulPartitionedCall_1")
  .feed("saver_filename", t.createTensor(variablesFile))
  .run()

val graphFile = Paths.get(exportDir, TensorflowWrapper.SavedModelPB).toString
FileUtils.writeByteArrayToFile(new File(graphFile), graph.toGraphDef.toByteArray)

We have been doing this since TF v1 and it has been compatible with all the TF v2 up until 0.4.0, but now if we save it like this (only models from TF Hub and I am guessing those with assets) we see that error)

Originally posted by @maziyarpanahi in https://github.com/tensorflow/java/issues/387#issuecomment-1066628248

Tái hiện lỗi bằng cách sử dụng SavedModelBundle.load, graph và session, cùng với đích saver_filename được hiển thị trong issue, đồng thời so sánh các model TF Hub và HuggingFace. Kiểm tra cách graph và các biến được lưu vào SavedModelPB rồi được tải lại. Hoàn tất khi các model TF Hub có assets có thể được lưu và tải lại mà không gặp lỗi asset_path_initializer.