"stornext" file system : too long file names lead to installation failures
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5.1k
- Forks
- 2.5k
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 82
Description
I tried to install a long list of dependent software on a cluster with CentOS7 using spack 0.16.0.
- The installation functions on a single Ubuntu-18.04 standalone system.
- I tried to install on the home directory partition, which is located on an external StorNext filesystem. Even though the concretization was completed successfully, the installation exited after hours of installing most of the software with a KeyError when I tried to install. The final messages are given below.
- I installed the software successfully on a BeeGFS partition.
Is there a problem using Spack on StorNext filesystems? What could be causing this and how can it be prevented?
$ spack -d -D /usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps install
...
==> [2021-01-12-13:13:06.997285] 'file' '-b' '-h' '/usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/linux-scientific7-haswell/gcc-4.8.5/py-sphinx-3.2.0-l5m4rjdm6dx6x7nufzbjq76nxqn7755m/bin/sphinx-build'
==> [2021-01-12-13:13:07.015073] FILTER FILE: /usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/.spack-env/view/bin/sphinx-build [replacing "/usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/linux-scientific7-haswell/gcc-4.8.5/python-3.7.7-jjmasvisx44bibq7lnuf4hp7frplyzqz"]
==> [2021-01-12-13:13:11.501396] 'file' '-b' '-h' '/usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/linux-scientific7-haswell/gcc-4.8.5/py-chardet-3.0.4-qsh6e725rhn6j4hakjhth6cqyrbulwan/bin/chardetect'
==> [2021-01-12-13:13:11.512019] FILTER FILE: /usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/.spack-env/view/bin/chardetect [replacing "/usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/linux-scientific7-haswell/gcc-4.8.5/python-3.7.7-jjmasvisx44bibq7lnuf4hp7frplyzqz"]
==> [2021-01-12-13:13:17.503420] 'file' '-b' '-h' '/usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/linux-scientific7-haswell/gcc-4.8.5/py-setuptools-50.3.2-3i76abgl6tldeyumhtwsoukx5wxitxwt/bin/easy_install'
==> [2021-01-12-13:13:17.520867] FILTER FILE: /usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/.spack-env/view/bin/easy_install [replacing "/usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/linux-scientific7-haswell/gcc-4.8.5/python-3.7.7-jjmasvisx44bibq7lnuf4hp7frplyzqz"]
==> [2021-01-12-13:13:17.750287] 'file' '-b' '-h' '/usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/linux-scientific7-haswell/gcc-4.8.5/py-setuptools-50.3.2-3i76abgl6tldeyumhtwsoukx5wxitxwt/bin/easy_install-3.7'
==> [2021-01-12-13:13:17.766679] FILTER FILE: /usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/.spack-env/view/bin/easy_install-3.7 [replacing "/usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/linux-scientific7-haswell/gcc-4.8.5/python-3.7.7-jjmasvisx44bibq7lnuf4hp7frplyzqz"]
==> [2021-01-12-13:13:24.331889] 'file' '-b' '-h' '/usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/linux-scientific7-haswell/gcc-4.8.5/py-future-0.17.1-ud6zrfv7qqvyh7p5iyikgvog2ngxu2vh/bin/futurize'
==> [2021-01-12-13:13:24.349016] FILTER FILE: /usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/.spack-env/view/bin/futurize [replacing "/usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/linux-scientific7-haswell/gcc-4.8.5/python-3.7.7-jjmasvisx44bibq7lnuf4hp7frplyzqz"]
==> [2021-01-12-13:13:24.373910] 'file' '-b' '-h' '/usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/linux-scientific7-haswell/gcc-4.8.5/py-future-0.17.1-ud6zrfv7qqvyh7p5iyikgvog2ngxu2vh/bin/pasteurize'
==> [2021-01-12-13:13:24.392665] FILTER FILE: /usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/.spack-env/view/bin/pasteurize [replacing "/usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/linux-scientific7-haswell/gcc-4.8.5/python-3.7.7-jjmasvisx44bibq7lnuf4hp7frplyzqz"]
==> [2021-01-12-13:14:31.004078] DIRECTORY RECOVERED [/usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/.spack-env/view]
Traceback (most recent call last):
File "/home/uni11/gwdg1/GWDG/rmeuer1/dlr/flowsimbuilder-reloaded/spack/lib/spack/llnl/util/filesystem.py", line 686, in replace_directory_transaction
yield tmp_dir
File "/home/uni11/gwdg1/GWDG/rmeuer1/dlr/flowsimbuilder-reloaded/spack/lib/spack/spack/environment.py", line 576, in regenerate
view.add_specs(*add_specs, with_dependencies=False)
File "/home/uni11/gwdg1/GWDG/rmeuer1/dlr/flowsimbuilder-reloaded/spack/lib/spack/spack/filesystem_view.py", line 306, in add_specs
all(map(self.add_extension, extensions))
File "/home/uni11/gwdg1/GWDG/rmeuer1/dlr/flowsimbuilder-reloaded/spack/lib/spack/spack/filesystem_view.py", line 321, in add_extension
self, verbose=self.verbose, with_dependencies=False)
File "/home/uni11/gwdg1/GWDG/rmeuer1/dlr/flowsimbuilder-reloaded/spack/lib/spack/spack/package.py", line 2299, in do_activate
self, view, **self.extendee_args)
File "/home/uni11/gwdg1/GWDG/rmeuer1/dlr/flowsimbuilder-reloaded/spack/var/spack/repos/builtin/packages/python/package.py", line 1067, in activate
File "/home/uni11/gwdg1/GWDG/rmeuer1/dlr/flowsimbuilder-reloaded/spack/lib/spack/spack/package.py", line 2322, in activate
view.merge(extension.spec, ignore=kwargs.get('ignore', None))
File "/home/uni11/gwdg1/GWDG/rmeuer1/dlr/flowsimbuilder-reloaded/spack/lib/spack/spack/filesystem_view.py", line 391, in merge
pkg.add_files_to_view(self, merge_map)
File "/home/uni11/gwdg1/GWDG/rmeuer1/dlr/flowsimbuilder-reloaded/spack/lib/spack/spack/build_systems/python.py", line 438, in add_files_to_view
new_link_target = os.path.abspath(merge_map[orig_link_target])
KeyError: '/home/uni11/gwdg1/GWDG/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/linux-scientific7-haswell/gcc-4.8.5/py-docutils-0.15.2-5wq7ft6wxzhnb6d3dclae2b2amjqmds5/bin/rst2latex.py'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/uni11/gwdg1/GWDG/rmeuer1/dlr/flowsimbuilder-reloaded/spack/bin/spack", line 66, in <module>
sys.exit(spack.main.main())
File "/home/uni11/gwdg1/GWDG/rmeuer1/dlr/flowsimbuilder-reloaded/spack/lib/spack/spack/main.py", line 762, in main
return _invoke_command(command, parser, args, unknown)
File "/home/uni11/gwdg1/GWDG/rmeuer1/dlr/flowsimbuilder-reloaded/spack/lib/spack/spack/main.py", line 490, in _invoke_command
return_val = command(parser, args)
File "/home/uni11/gwdg1/GWDG/rmeuer1/dlr/flowsimbuilder-reloaded/spack/lib/spack/spack/cmd/install.py", line 259, in install
env.install_all(args, **kwargs)
File "/home/uni11/gwdg1/GWDG/rmeuer1/dlr/flowsimbuilder-reloaded/spack/lib/spack/spack/environment.py", line 1452, in install_all
self.regenerate_views()
File "/home/uni11/gwdg1/GWDG/rmeuer1/dlr/flowsimbuilder-reloaded/spack/lib/spack/spack/environment.py", line 1227, in regenerate_views
view.regenerate(specs, self.roots())
File "/home/uni11/gwdg1/GWDG/rmeuer1/dlr/flowsimbuilder-reloaded/spack/lib/spack/spack/environment.py", line 576, in regenerate
view.add_specs(*add_specs, with_dependencies=False)
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/home/uni11/gwdg1/GWDG/rmeuer1/dlr/flowsimbuilder-reloaded/spack/lib/spack/llnl/util/filesystem.py", line 699, in replace_directory_transaction
raise RuntimeError(msg.format(directory_name))
RuntimeError: the transactional move of "/usr/users/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/.spack-env/view" failed.
'/home/uni11/gwdg1/GWDG/rmeuer1/local/dlr/flowsim_gnu-eigen2-deps/linux-scientific7-haswell/gcc-4.8.5/py-docutils-0.15.2-5wq7ft6wxzhnb6d3dclae2b2amjqmds5/bin/rst2latex.py'
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported spack -d -D ... install command on the StorNext filesystem and read the traceback through lib/spack/spack/build_systems/python.py, lib/spack/spack/spack/filesystem_view.py, and lib/spack/spack/spack/environment.py. Trace the missing rst2latex.py path during view regeneration; done means the installation no longer fails with the reported KeyError and transactional-move error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100