DELG export model problem

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• I am reporting the issue to the correct repository. (Model Garden official or research directory)
• I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/blob/master/research/delf/delf/python/training/README.md

2. Describe the bug

Getting errors when testing the trained DELF model.
After run the following commands:
python3 ../examples/extract_features.py \ --config_path delf_config_example.pbtxt \ --list_images_path list_images.txt \ --output_dir data/oxford5k_features
I get some error logs:
`Reading list of images...
done! Found 2 images
2022-02-15 18:54:47.168388: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-02-15 18:54:47.589301: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9834 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3080 Ti, pci bus id: 0000:65:00.0, compute capability: 8.6
Starting to extract DELF features from images...
2022-02-15 18:54:54.206525: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8100
Traceback (most recent call last):
File "../examples/extract_features.py", line 142, in
app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/home/kmj-lab/anaconda3/envs/py3.8/lib/python3.8/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/home/kmj-lab/anaconda3/envs/py3.8/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "../examples/extract_features.py", line 105, in main
extracted_features = extractor_fn(im)
File "/mnt/sdb/research/delf/delf/python/examples/extractor.py", line 195, in ExtractorFn
output_dict = predict(
File "/home/kmj-lab/anaconda3/envs/py3.8/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1707, in call
return self._call_impl(args, kwargs)
File "/home/kmj-lab/anaconda3/envs/py3.8/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1716, in _call_impl
return self._call_with_structured_signature(args, kwargs,
File "/home/kmj-lab/anaconda3/envs/py3.8/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1797, in _call_with_structured_signature
return self._call_flat(
File "/home/kmj-lab/anaconda3/envs/py3.8/lib/python3.8/site-packages/tensorflow/python/saved_model/load.py", line 122, in _call_flat
return super(_WrapperFunction, self)._call_flat(args, captured_inputs,
File "/home/kmj-lab/anaconda3/envs/py3.8/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1959, in _call_flat
return self._build_call_outputs(self._inference_function.call(
File "/home/kmj-lab/anaconda3/envs/py3.8/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 598, in call
outputs = execute.execute(
File "/home/kmj-lab/anaconda3/envs/py3.8/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 58, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 50048 values, but the requested shape requires a multiple of 1024
[[node while/Reshape_2
(defined at /mnt/sdb/research/delf/delf/python/examples/extractor.py:52)
]] [Op:__inference_signature_wrapper_14736]

Errors may have originated from an input operation.
Input Source operations connected to node while/Reshape_2:
In[0] while/Squeeze_1:
In[1] while/Reshape_2/shape:

Operation defined at: (most recent call last)

File "../examples/extract_features.py", line 142, in
app.run(main=main, argv=[sys.argv[0]] + unparsed)

File "/home/kmj-lab/anaconda3/envs/py3.8/lib/python3.8/site-packages/absl/app.py", line 312, in run
_run_main(main, args)

File "/home/kmj-lab/anaconda3/envs/py3.8/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))

File "../examples/extract_features.py", line 81, in main
extractor_fn = extractor.MakeExtractor(config)

File "/mnt/sdb/research/delf/delf/python/examples/extractor.py", line 52, in MakeExtractor
model = tf.saved_model.load(config.model_path)

Function call stack:
signature_wrapper -> while_body_6754

`

3. Steps to reproduce

I followed the code on DELF/DELG Training Instructions.
This is what I did:

• Training with Local and Global Features
• Exporting DELF local feature-only model
• Following instructions to change the delf_config_example.pbtxt
• Then I extract DELF features for the images hertford_000056.jpg and oxford_000317.jpg and get the error logs above.

4. Expected behavior

I really appriciate for any help on this issue. I have no idea how to solve this.

5. Additional context

If needed, when I run python3 model/export_local_model.py \ --ckpt_path=gldv2_training/delf_weights \ --export_path=gldv2_model_local
I get logs:
2022-02-15 16:57:20.843683: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2022-02-15 16:57:21.214404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9843 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3080 Ti, pci bus id: 0000:65:00.0, compute capability: 8.6 Checkpoint loaded from gldv2_training/delf_weights 2022-02-15 16:57:21.465479: W tensorflow/python/util/util.cc:368] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them. WARNING:tensorflow:Skipping full serialization of Keras layer <delf.python.training.model.resnet50.ResNet50 object at 0x7f25510aa700>, because it is not built. W0215 16:57:24.956270 139803146138560 save_impl.py:71] Skipping full serialization of Keras layer <delf.python.training.model.resnet50.ResNet50 object at 0x7f25510aa700>, because it is not built. WARNING:tensorflow:Skipping full serialization of Keras layer <keras.layers.pooling.AveragePooling2D object at 0x7f25483efdc0>, because it is not built. W0215 16:57:29.897069 139803146138560 save_impl.py:71] Skipping full serialization of Keras layer <keras.layers.pooling.AveragePooling2D object at 0x7f25483efdc0>, because it is not built. W0215 16:57:34.503767 139803146138560 save.py:263] Found untraced functions such as conv1_layer_call_fn, conv1_layer_call_and_return_conditional_losses, conv1_layer_call_fn, conv1_layer_call_and_return_conditional_losses, conv1_layer_call_and_return_conditional_losses while saving (showing 5 of 5). These functions will not be directly callable after loading. INFO:tensorflow:Assets written to: gldv2_model_local/assets I0215 16:57:36.854881 139803146138560 builder_impl.py:783] Assets written to: gldv2_model_local/assets WARNING:tensorflow:Unresolved object in checkpoint: (root).cosine_weights W0215 16:57:37.276778 139803146138560 util.py:181] Unresolved object in checkpoint: (root).cosine_weights WARNING:tensorflow:Unresolved object in checkpoint: (root).scale_factor W0215 16:57:37.276922 139803146138560 util.py:181] Unresolved object in checkpoint: (root).scale_factor WARNING:tensorflow:Unresolved object in checkpoint: (root).attn_classification W0215 16:57:37.276959 139803146138560 util.py:181] Unresolved object in checkpoint: (root).attn_classification WARNING:tensorflow:Unresolved object in checkpoint: (root).backbone.embedding_layer W0215 16:57:37.276995 139803146138560 util.py:181] Unresolved object in checkpoint: (root).backbone.embedding_layer WARNING:tensorflow:Unresolved object in checkpoint: (root).attn_classification.kernel W0215 16:57:37.277029 139803146138560 util.py:181] Unresolved object in checkpoint: (root).attn_classification.kernel WARNING:tensorflow:Unresolved object in checkpoint: (root).attn_classification.bias W0215 16:57:37.277061 139803146138560 util.py:181] Unresolved object in checkpoint: (root).attn_classification.bias WARNING:tensorflow:Unresolved object in checkpoint: (root).backbone.embedding_layer.kernel W0215 16:57:37.277096 139803146138560 util.py:181] Unresolved object in checkpoint: (root).backbone.embedding_layer.kernel WARNING:tensorflow:Unresolved object in checkpoint: (root).backbone.embedding_layer.bias W0215 16:57:37.277127 139803146138560 util.py:181] Unresolved object in checkpoint: (root).backbone.embedding_layer.bias WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details. W0215 16:57:37.277177 139803146138560 util.py:189] A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04
• Mobile device name if the issue happens on a mobile device:
• TensorFlow installed from (source or binary):
• TensorFlow version (use command below): 2.7.0
• Python version: 3.8.12
• Bazel version (if compiling from source):
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version: CUDA 11.2+cuDNN 8.1
• GPU model and memory: RTX 3080 Ti