tensorflow / tensorflow/models

SegFault when trying to train DeepLabv3+ on cityscapes train_fine with xception_65

Open
#8,365 1 comment 0 reactions 1 assignee View on GitHub

@aquariusjay is already working on this.

Since May 8, 2020.

models:research type:bug
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

In an attempt to reproduce the examples in https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/cityscapes.md to train a segmentation model using cityscapes training set (train_fine) I am getting a Segmentation Fault error when running:

python deeplab/train.py --logtostderr --training_number_of_steps=90000 --train_split="train_fine" --model_variant="xception_65" --atrous_rates=6 --atrous_rates=12 --atrous_rates=18 --output_stride=16 --decoder_output_stride=4 --train_crop_size="769,769" --train_batch_size=1 --dataset="cityscapes" --train_logdir=${PATH_TO_LOG_DIR} --dataset_dir=${PATH_TO_DATASET_DIR_POINTING_TO_TFRECORD}

The segFault message is:

INFO:tensorflow:Running local_init_op.
I0406 10:01:29.072682 140678180361984 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0406 10:01:29.720130 140678180361984 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Starting Session.
I0406 10:01:37.946371 140678180361984 learning.py:754] Starting Session.
INFO:tensorflow:Saving checkpoint to path ${PATH_TO_LOG_DIR}/model.ckpt
I0406 10:01:38.327494 140673365833472 supervisor.py:1117] Saving checkpoint to path ${PATH_TO_LOG_DIR}/model.ckpt
INFO:tensorflow:Starting Queues.
I0406 10:01:38.327682 140678180361984 learning.py:768] Starting Queues.
INFO:tensorflow:global_step/sec: 0
I0406 10:01:41.273570 140673617483520 supervisor.py:1099] global_step/sec: 0
Fatal Python error: Segmentation fault

Thread 0x00007ff111fff700 (most recent call first):
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3175 in _as_graph_def
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3238 in as_graph_def
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1246 in export_meta_graph
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1203 in save
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/supervisor.py", line 1119 in run_loop
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/coordinator.py", line 495 in run
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 926 in _bootstrap_inner
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007ff120ffd700 (most recent call first):
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 300 in wait
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 552 in wait
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/coordinator.py", line 311 in wait_for_stop
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/coordinator.py", line 493 in run
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 926 in _bootstrap_inner
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007ff1217fe700 (most recent call first):
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443 in _call_tf_sessionrun
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350 in _run_fn
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365 in _do_call
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359 in _do_run
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180 in _run
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956 in run
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/supervisor.py", line 1045 in run_loop
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/coordinator.py", line 495 in run
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 926 in _bootstrap_inner
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007ff1b67fc700 (most recent call first):
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 296 in wait
File "/home/grupo07/anaconda3/lib/python3.7/queue.py", line 170 in get
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/summary/writer/event_file_writer.py", line 159 in run
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 926 in _bootstrap_inner
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007ff230f7d700 (most recent call first):
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443 in _call_tf_sessionrun
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350 in _run_fn
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365 in _do_call
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359 in _do_run
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180 in _run
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956 in run
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/contrib/slim/python/slim/learning.py", line 490 in train_step
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/contrib/slim/python/slim/learning.py", line 775 in train
File "deeplab/train.py", line 458 in main
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/absl/app.py", line 250 in _run_main
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/absl/app.py", line 299 in run
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/platform/app.py", line 40 in run
File "deeplab/train.py", line 464 in
/var/slurm/SlurmdSpoolDir/job06688/slurm_script: línea 22: 35771 Violación de segmento (`core' generado) python deeplab/train.py --logtostderr --training_number_of_steps=90000 --train_split="train_fine" --model_variant="xception_65" --atrous_rates=6 --atrous_rates=12 --atrous_rates=18 --output_stride=16 --decoder_output_stride=4 --train_crop_size="769,769" --train_batch_size=1 --dataset="cityscapes" --train_logdir=${PATH_TO_LOG_DIR} --dataset_dir=${PATH_TO_DATASET_DIR_POINTING_TO_TFRECORD}

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

Used example script from documentation

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

Linux Ubuntu 16.04.6 LTS (GNU/Linux 4.4.0-171-generic x86_64)

  • TensorFlow installed from (source or binary):

from binary

  • TensorFlow version (use command below):

1.15 (2.0 was not working)

  • Python version:

Tried with 3.7 and 2.7

  • CUDA/cuDNN version:

CUDA Version 10.2.89

  • GPU model and memory:

TITAN X (Pascal) 12G

Describe the current behavior
Obtaining Segmentation Fault

Describe the expected behavior
Train working successfully

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.