tensorflow / tensorflow/models

No OpKernel was registered to support Op 'TPUOrdinalSelector' used by node TPUOrdinalSelector

Open
#7,501 9 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

models:research
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

System information
  • What is the top-level directory of the model you are using:
  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04
  • TensorFlow installed from (source or binary): have tried both, first binary install and then from source
  • TensorFlow version (use command below): 1.14
  • Bazel version (if compiling from source): 0.26.1
  • CUDA/cuDNN version:10.0.130
  • GPU model and memory: NVIDIA GeForce GTX 1080 8GB Ram
  • Exact command to reproduce: python3 object_detection/tpu_exporters/export_saved_model_tpu.py

I have trained a model on my GPU and want to export the model to a TPU for inference.

I am running the script found at

models/research/object_detection/tpu_exporters/export_saved_model_tpu.py

When I run the below command:
python3 object_detection/tpu_exporters/export_saved_model_tpu.py --pipeline_config_file=object_detection/output/models/model/ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync.config --ckpt_path=object_detection/output/models/model/model.ckpt-39688 --export_dir=/tmp/out --input_type="image_tensor" --input_placeholder_name=image_tensor:0
I get the following exception:

With the full exception being 
Traceback (most recent call last):
  File "/home/johnconn/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/home/johnconn/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1348, in _run_fn
    self._extend_graph()
  File "/home/johnconn/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1388, in _extend_graph
    tf_session.ExtendSession(self._session)
tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'TPUOrdinalSelector' used by {{node TPUOrdinalSelector}}with these attrs: []
Registered devices: [CPU, GPU]
Registered kernels:
  <no registered kernels>

	 [[TPUOrdinalSelector]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "object_detection/tpu_exporters/export_saved_model_tpu.py", line 54, in <module>
    tf.app.run()
  File "/home/johnconn/.local/lib/python3.6/site-packages/tensorflow_core/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/home/johnconn/.local/lib/python3.6/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/home/johnconn/.local/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "object_detection/tpu_exporters/export_saved_model_tpu.py", line 47, in main
    FLAGS.input_type, FLAGS.use_bfloat16)
  File "/home/johnconn/utils/models/research/object_detection/tpu_exporters/export_saved_model_tpu_lib.py", line 80, in export
    sess.run(init_op)
  File "/home/johnconn/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/home/johnconn/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/johnconn/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/home/johnconn/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'TPUOrdinalSelector' used by node TPUOrdinalSelector (defined at /home/johnconn/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) with these attrs: []
Registered devices: [CPU, GPU]
Registered kernels:
  <no registered kernels>

	 [[TPUOrdinalSelector]]

It looks like @shreyaaggarwal encountered the same issue, posting it here https://github.com/tensorflow/models/issues/4283

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with models/research/object_detection/tpu_exporters/export_saved_model_tpu.py and export_saved_model_tpu_lib.py, then reproduce the command using the reported TensorFlow 1.14 setup. Determine the expected behavior when only CPU and GPU devices are registered, and make the export path either work in that environment or clearly report its TPU requirements.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.