NVIDIA / NVIDIA/hpc-container-maker

Singularity %files behavior change in 3.6 and later

Open
#345 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
517
Forks
106
Avg merge
1d 22h
Merged PRs (30d)
2

Description

The behavior of the Singularity %files directive changed beginning in version 3.6. See https://github.com/hpcng/singularity/issues/5514.

For this Singularity definition file:

Bootstrap: docker
From: ubuntu:18.04

%files
   foo /var/tmp

%post
   ls -l /var/tmp/foo

Singularity 3.5.3:

# singularity --version
singularity version 3.5.3

# singularity build foo.sif Singularity.def 
INFO:    Starting build...
INFO:    Copying foo to /tmp/rootfs-f4ee8c55-7790-11eb-b63f-0242ac110002/var/tmp
INFO:    Running post scriptlet
+ ls -l /var/tmp/foo
-rw-r--r--. 1 root root 0 Feb 25 17:43 /var/tmp/foo
INFO:    Creating SIF file...
INFO:    Build complete: foo.sif

Singularity 3.6.4:

# singularity --version
singularity version 3.6.4

# singularity build foo.sif Singularity.def 
INFO:    Starting build...
INFO:    Copying foo to /tmp/build-temp-392680558/rootfs/var/tmp
INFO:    Running post scriptlet
+ ls -l /var/tmp/foo
ls: cannot access '/var/tmp/foo': No such file or directory

Changing the path from /var/tmp to /opt works with either version:

INFO:    Copying foo to /tmp/build-temp-697021000/rootfs/opt
INFO:    Running post scriptlet
+ ls -l /opt/foo
-rw-r--r--. 1 root root 0 Feb 25 17:41 /opt/foo
INFO:    Creating SIF file...
INFO:    Build complete: foo.sif

The --working-directory=/other/path option can be used to generate a Singularity definition file that works around this behavior change.

A better solution might be to use %setup to stage files into the container build environment. Alternatively, use a default location other than /var/tmp or /tmp when copying files from the host (such as the package option in many building blocks).

Originally reported in #341 by @VI-gha

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no repository files, tests, or entry points. Start by reproducing the shown Singularity 3.5.3 and 3.6.4 builds with the example definition, then trace how host files and destinations are generated. Done should mean the generated definition handles the reported /var/tmp behavior across the affected Singularity versions without regressing the existing /opt case.

Written by the indexing model from the issue text.

Assessment

Domain
build-system, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.