starting container process caused 'process_linux.go:245: running exec setns process for init caused "exit status 6"'

Hi OCI folks,

We are seeing a failure to start Docker containers through runc, seemingly from this line:

• https://github.com/opencontainers/runc/blob/master/libcontainer/process_linux.go#L245

This might well be a config or system issue (we're on somewhat old Kernel versions because CentOS..), but the logs don't give so much to go on here..

The man pages for setns is defining the error codes it should return:

• http://man7.org/linux/man-pages/man2/setns.2.html

But if the following page can be trusted, exit status 6 should be ENXIO, which is not mentioned in the man pages:

• http://www.virtsync.com/c-error-codes-include-errno

Any suggestions for how to debug further or what to check would be appreciated, thanks in advance!

Logs

/bin/docker: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:245: running exec setns process for init caused \\\"exit status 6\\\"\"\n".

System info

# uname -a
Linux ip-10-226-24-78 3.10.0-327.28.2.el7.x86_64 #1 SMP Wed Aug 3 11:11:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

# docker info
Containers: 1
 Running: 1
  Paused: 0
     Stopped: 0
     Images: 6
     Server Version: 1.12.2
     Storage Driver: overlay
      Backing Filesystem: xfs
        Logging Driver: json-file
        Cgroup Driver: cgroupfs
        Plugins:
         Volume: local
          Network: bridge null host overlay
            Swarm: inactive
            Runtimes: runc
            Default Runtime: runc
            Security Options: seccomp
            Kernel Version: 3.10.0-327.28.2.el7.x86_64
            Operating System: CentOS Linux 7 (Core)
            OSType: linux
            Architecture: x86_64
            CPUs: 2
            Total Memory: 7.389 GiB
            Name: ip-10-226-24-78
            ID: TNS5:V674:K6Y4:CSIT:ROPR:XJMI:LDSR:KTC3:DZS7:G7RD:426H:DFRN
            Docker Root Dir: /var/lib/docker
            Debug Mode (client): false
            Debug Mode (server): false
            Registry: https://index.docker.io/v1/
            WARNING: bridge-nf-call-iptables is disabled
            WARNING: bridge-nf-call-ip6tables is disabled
            Insecure Registries:
             127.0.0.0/8

# free -m
              total        used        free      shared  buff/cache   available
Mem:           7566         207         453           5        6904        4230
Swap:          2047         463        1584

# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    1
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
Stepping:              2
CPU MHz:               2400.082
BogoMIPS:              4800.16
Hypervisor vendor:     Xen
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              30720K
NUMA node0 CPU(s):     0,1

Start at libcontainer/process_linux.go:245 and compare the reported runc failure with the setns behavior on the provided CentOS 7, kernel 3.10, and Docker 1.12.2 environment. Reproduce the container-start failure and identify what produces exit status 6; done means documenting a confirmed cause and actionable debugging or remediation steps.