tensorflow / tensorflow/models
Error submitting Object Detection job to ai-platform
@pkulzc is already working on this.
Since Jan 25, 2021.
- Dominant language
- Python
- Stars
- 77.7k
- Forks
- 44.8k
- PR merge metrics
- No merged PRs in 30d
Description
I am using the instructions laid out in the documentation to submit a job to Google Clouds ai-platform. My pipeline runs correctly both locally and when deploying to a Google Cloud VM.
The command I am running is below, it is practically the same as the documentation, except updated for the latest runtime versions. I have also tried the GPU training instructions with the same result
cp object_detection/packages/tf2/setup.py .
gcloud ai-platform jobs submit training `whoami`_object_detection_`date +%m_%d_%Y_%H_%M_%S` \
--job-dir=${MODEL_DIR} \
--package-path ./object_detection \
--module-name object_detection.model_main_tf2 \
--runtime-version 2.3 \
--python-version 3.7 \
--scale-tier BASIC_TPU \
--region us-central1 \
-- \
--use_tpu true \
--model_dir=${MODEL_DIR} \
--pipeline_config_path=${PIPELINE_CONFIG_PATH}
In the AI Platform job details I am receiving the error message below. I have seen this error brought up in a couple of other places, including here. But nothing is referenced, which it would appear I have the ability to configure in the context of submitting a job to the ai-platform
The replica master 0 exited with a non-zero status of 1.
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/root/.local/lib/python3.7/site-packages/object_detection/model_main_tf2.py", line 31, in <module>
import tensorflow.compat.v2 as tf
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
File "/opt/conda/lib/python3.7/site-packages/wrapt/importer.py", line 156, in load_module
module = self.loader.load_module(fullname)
File "/root/.local/lib/python3.7/site-packages/tensorflow/__init__.py", line 436, in <module>
_ll.load_library(_main_dir)
File "/root/.local/lib/python3.7/site-packages/tensorflow/python/framework/load_library.py", line 153, in load_library
py_tf.TF_LoadLibrary(lib)
tensorflow.python.framework.errors_impl.NotFoundError: /opt/conda/lib/python3.7/site-packages/tensorflow/core/kernels/libtfkernel_sobol_op.so: undefined symbol: _ZN10tensorflow8OpKernel11TraceStringEPNS_15OpKernelContextEb
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.