tensorflow / tensorflow/models
SegFault when trying to train DeepLabv3+ on cityscapes train_fine with xception_65
@aquariusjay is already working on this.
Since May 8, 2020.
- Dominant language
- Python
- Stars
- 77.7k
- Forks
- 44.8k
- PR merge metrics
- No merged PRs in 30d
Description
In an attempt to reproduce the examples in https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/cityscapes.md to train a segmentation model using cityscapes training set (train_fine) I am getting a Segmentation Fault error when running:
python deeplab/train.py
--logtostderr
--training_number_of_steps=90000
--train_split="train_fine"
--model_variant="xception_65"
--atrous_rates=6
--atrous_rates=12
--atrous_rates=18
--output_stride=16
--decoder_output_stride=4
--train_crop_size="769,769"
--train_batch_size=1
--dataset="cityscapes"
--train_logdir=${PATH_TO_LOG_DIR}
--dataset_dir=${PATH_TO_DATASET_DIR_POINTING_TO_TFRECORD}
The segFault message is:
INFO:tensorflow:Running local_init_op.
I0406 10:01:29.072682 140678180361984 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0406 10:01:29.720130 140678180361984 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Starting Session.
I0406 10:01:37.946371 140678180361984 learning.py:754] Starting Session.
INFO:tensorflow:Saving checkpoint to path ${PATH_TO_LOG_DIR}/model.ckpt
I0406 10:01:38.327494 140673365833472 supervisor.py:1117] Saving checkpoint to path ${PATH_TO_LOG_DIR}/model.ckpt
INFO:tensorflow:Starting Queues.
I0406 10:01:38.327682 140678180361984 learning.py:768] Starting Queues.
INFO:tensorflow:global_step/sec: 0
I0406 10:01:41.273570 140673617483520 supervisor.py:1099] global_step/sec: 0
Fatal Python error: Segmentation fault
Thread 0x00007ff111fff700 (most recent call first):
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3175 in _as_graph_def
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3238 in as_graph_def
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1246 in export_meta_graph
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1203 in save
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/supervisor.py", line 1119 in run_loop
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/coordinator.py", line 495 in run
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 926 in _bootstrap_inner
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 890 in _bootstrapThread 0x00007ff120ffd700 (most recent call first):
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 300 in wait
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 552 in wait
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/coordinator.py", line 311 in wait_for_stop
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/coordinator.py", line 493 in run
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 926 in _bootstrap_inner
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 890 in _bootstrapThread 0x00007ff1217fe700 (most recent call first):
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443 in _call_tf_sessionrun
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350 in _run_fn
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365 in _do_call
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359 in _do_run
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180 in _run
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956 in run
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/supervisor.py", line 1045 in run_loop
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/coordinator.py", line 495 in run
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 926 in _bootstrap_inner
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 890 in _bootstrapThread 0x00007ff1b67fc700 (most recent call first):
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 296 in wait
File "/home/grupo07/anaconda3/lib/python3.7/queue.py", line 170 in get
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/summary/writer/event_file_writer.py", line 159 in run
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 926 in _bootstrap_inner
File "/home/grupo07/anaconda3/lib/python3.7/threading.py", line 890 in _bootstrapThread 0x00007ff230f7d700 (most recent call first):
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443 in _call_tf_sessionrun
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350 in _run_fn
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365 in _do_call
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359 in _do_run
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180 in _run
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956 in run
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/contrib/slim/python/slim/learning.py", line 490 in train_step
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/contrib/slim/python/slim/learning.py", line 775 in train
File "deeplab/train.py", line 458 in main
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/absl/app.py", line 250 in _run_main
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/absl/app.py", line 299 in run
File "/home/grupo07/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/platform/app.py", line 40 in run
File "deeplab/train.py", line 464 in
/var/slurm/SlurmdSpoolDir/job06688/slurm_script: línea 22: 35771 Violación de segmento (`core' generado) python deeplab/train.py --logtostderr --training_number_of_steps=90000 --train_split="train_fine" --model_variant="xception_65" --atrous_rates=6 --atrous_rates=12 --atrous_rates=18 --output_stride=16 --decoder_output_stride=4 --train_crop_size="769,769" --train_batch_size=1 --dataset="cityscapes" --train_logdir=${PATH_TO_LOG_DIR} --dataset_dir=${PATH_TO_DATASET_DIR_POINTING_TO_TFRECORD}
System information
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
Used example script from documentation
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Linux Ubuntu 16.04.6 LTS (GNU/Linux 4.4.0-171-generic x86_64)
- TensorFlow installed from (source or binary):
from binary
- TensorFlow version (use command below):
1.15 (2.0 was not working)
- Python version:
Tried with 3.7 and 2.7
- CUDA/cuDNN version:
CUDA Version 10.2.89
- GPU model and memory:
TITAN X (Pascal) 12G
Describe the current behavior
Obtaining Segmentation Fault
Describe the expected behavior
Train working successfully
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.