tensorflow / tensorflow/models

Error submitting Object Detection job to ai-platform

Open
#9,658 6 comments 1 reaction 3 assignees View on GitHub

@pkulzc is already working on this.

Since Jan 25, 2021.

models:research:odapi type:bug
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

I am using the instructions laid out in the documentation to submit a job to Google Clouds ai-platform. My pipeline runs correctly both locally and when deploying to a Google Cloud VM.

The command I am running is below, it is practically the same as the documentation, except updated for the latest runtime versions. I have also tried the GPU training instructions with the same result

cp object_detection/packages/tf2/setup.py .
gcloud ai-platform jobs submit training `whoami`_object_detection_`date +%m_%d_%Y_%H_%M_%S` \
    --job-dir=${MODEL_DIR} \
    --package-path ./object_detection \
    --module-name object_detection.model_main_tf2 \
    --runtime-version 2.3 \
    --python-version 3.7 \
    --scale-tier BASIC_TPU \
    --region us-central1 \
    -- \
    --use_tpu true \
    --model_dir=${MODEL_DIR} \
    --pipeline_config_path=${PIPELINE_CONFIG_PATH}

In the AI Platform job details I am receiving the error message below. I have seen this error brought up in a couple of other places, including here. But nothing is referenced, which it would appear I have the ability to configure in the context of submitting a job to the ai-platform

The replica master 0 exited with a non-zero status of 1. 
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/root/.local/lib/python3.7/site-packages/object_detection/model_main_tf2.py", line 31, in <module>
    import tensorflow.compat.v2 as tf
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/opt/conda/lib/python3.7/site-packages/wrapt/importer.py", line 156, in load_module
    module = self.loader.load_module(fullname)
  File "/root/.local/lib/python3.7/site-packages/tensorflow/__init__.py", line 436, in <module>
    _ll.load_library(_main_dir)
  File "/root/.local/lib/python3.7/site-packages/tensorflow/python/framework/load_library.py", line 153, in load_library
    py_tf.TF_LoadLibrary(lib)
tensorflow.python.framework.errors_impl.NotFoundError: /opt/conda/lib/python3.7/site-packages/tensorflow/core/kernels/libtfkernel_sobol_op.so: undefined symbol: _ZN10tensorflow8OpKernel11TraceStringEPNS_15OpKernelContextEb

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.