carpedm20 / carpedm20/NTM-tensorflow

Restore called with invalid save path

Open
#19 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1k
Forks
212
PR merge metrics
No merged PRs in 30d

Description

When I run the test mode

$ python main.py --task copy --test_max_length 120 --max_length 130

it shows the error "Restore called with invalid save path" but the path exists! Anyone else encountered the same problem?

I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally
{'checkpoint_dir': 'checkpoint',
'continue_train': None,
'controller_dim': 100,
'controller_layer_size': 1,
'epoch': 100000,
'input_dim': 10,
'is_train': False,
'max_length': 130,
'min_length': 1,
'output_dim': 10,
'read_head_size': 1,
'task': 'copy',
'test_max_length': 120,
'write_head_size': 1}
changepoint/copy_130
ln: failed to create symbolic link ‘copy_130/copy_10’: File exists
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties:
name: GeForce GTX 960
major: 5 minor: 2 memoryClockRate (GHz) 1.253
pciBusID 0000:01:00.0
Total memory: 3.94GiB
Free memory: 229.88MiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0: Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 960, pci bus id: 0000:01:00.0)
[*] Building a NTM model
Percent: [####################] 100.00% Finished.
[*] Build a NTM model finished
[*] Reading checkpoints...
Traceback (most recent call last):
File "/home/user/Desktop/NTM-tensorflow-master/main.py", line 83, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "/home/user/Desktop/NTM-tensorflow-master/main.py", line 70, in main
ntm.load(FLAGS.checkpoint_dir, FLAGS.task)
File "/home/user/Desktop/NTM-tensorflow-master/ntm.py", line 266, in load
self.saver.restore(self.sess, os.path.join(checkpoint_dir, ckpt_name))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1342, in restore
"File path is: %r" % (save_path, file_path))
ValueError: Restore called with invalid save path: u'checkpoint/copy_130/NTM-copy_copy.model-3402'. File path is: u'checkpoint/copy_130/NTM-copy_copy.model-3402'
[Finished in 286.5s with exit code 1]

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the checkpoint-loading call in ntm.py line 266 and its caller in main.py line 70. Inspect the files under checkpoint/copy_130 and compare them with the checkpoint name being passed to restore. Done means test mode can load the existing checkpoint without the invalid save path error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.