fatal msg="docker-slim: failure" error="wait timeout"

Hi,

I try to build a Centos 6.10 slim image to understand the process of Docker slim. All is run in docker in docker image from Gitlab CI. I unfortunately keep running into the following error which look like #94 :

docker-slim[build]: info=image.stack index=0 name='centos:6.10' id='sha256:48650444e4191276a912ac90816a4ce1e86f76419e004d445a2e42809fec05f7'
docker-slim[build]: info=image.stack index=1 name='centos-6.10:latest' id='sha256:e0b2b98132e9bd116ad47d76a5ce1f15af75a84feffab6d5e182d325eb997df4'
docker-slim[build]: state=image.inspection.done
docker-slim[build]: state=container.inspection.start
time="2020-02-11T19:45:16Z" level=debug msg="HasVolume: volume not found - docker-slim-sensor.1.26.1"
time="2020-02-11T19:45:16Z" level=debug msg="ensureSensorVolume: no volume yet = docker-slim-sensor.1.26.1" app=docker-slim command=build component=container.inspector
time="2020-02-11T19:45:16Z" level=debug msg="HasEmptyImage: empty image not found"
time="2020-02-11T19:45:17Z" level=debug msg="CreateVolumeWithData: volumeInfo = &{Name:docker-slim-sensor.1.26.1 Driver:local Mountpoint:/var/lib/docker/volumes/docker-slim-sensor.1.26.1/_data Labels:map[] Options:map[] CreatedAt:2020-02-11 19:45:17 +0000 UTC}"
time="2020-02-11T19:45:17Z" level=debug msg="CopyToVolume: containerID - 097faa00d72ce828374c4dd3c5f23fd36c434967b977a72e29d221944ef1f9e5"
time="2020-02-11T19:45:17Z" level=debug msg="version.Check: http status = 200 OK" app=docker-slim
time="2020-02-11T19:45:18Z" level=debug msg="RunContainer: default exposed ports => map[docker.Port]struct {}{\"65501/tcp\":struct {}{}, \"65502/tcp\":struct {}{}}" app=docker-slim command=build component=container.inspector
docker-slim[build]: info=container status=created name=dockerslimk_49_20200211194518 id=16cbabfebfeac9a9cea8073a3972a54a25c28a6bfe8a387741e52f397e9686ae
time="2020-02-11T19:45:19Z" level=debug msg="RunContainer: container NetworkSettings.Ports => map[docker.Port][]docker.PortBinding{\"65501/tcp\":[]docker.PortBinding{docker.PortBinding{HostIP:\"0.0.0.0\", HostPort:\"32769\"}}, \"65502/tcp\":[]docker.PortBinding{docker.PortBinding{HostIP:\"0.0.0.0\", HostPort:\"32768\"}}}" app=docker-slim command=build component=container.inspector
time="2020-02-11T19:45:19Z" level=debug msg="ipc.NewClient(172.18.0.2,65501,65502)"
time="2020-02-11T19:45:19Z" level=debug msg="channel.NewClient: connect wait timeout - 1m0s"
time="2020-02-11T19:45:19Z" level=debug msg="channel.NewClient: net.DialTimeout(tcp,172.18.0.2:65501,15s) [time=1581450319020594752]"
time="2020-02-11T19:45:34Z" level=debug msg="channel.NewClient: (dial time = 15.000224607s) - connect error = dial tcp 172.18.0.2:65501: i/o timeout"
time="2020-02-11T19:45:34Z" level=debug msg="channel.NewClient: waiting before trying to connect again..."
time="2020-02-11T19:45:36Z" level=debug msg="channel.NewClient: net.DialTimeout(tcp,172.18.0.2:65501,15s) [time=1581450336021039306]"
time="2020-02-11T19:45:51Z" level=debug msg="channel.NewClient: (dial time = 15.000283198s) - connect error = dial tcp 172.18.0.2:65501: i/o timeout"
time="2020-02-11T19:45:51Z" level=debug msg="channel.NewClient: waiting before trying to connect again..."
time="2020-02-11T19:45:53Z" level=debug msg="channel.NewClient: net.DialTimeout(tcp,172.18.0.2:65501,15s) [time=1581450353021560509]"
time="2020-02-11T19:46:08Z" level=debug msg="channel.NewClient: (dial time = 15.000244903s) - connect error = dial tcp 172.18.0.2:65501: i/o timeout"
time="2020-02-11T19:46:08Z" level=debug msg="channel.NewClient: waiting before trying to connect again..."
time="2020-02-11T19:46:10Z" level=debug msg="channel.NewClient: net.DialTimeout(tcp,172.18.0.2:65501,15s) [time=1581450370021993733]"
time="2020-02-11T19:46:25Z" level=debug msg="channel.NewClient: (dial time = 15.000259497s) - connect error = dial tcp 172.18.0.2:65501: i/o timeout"
time="2020-02-11T19:46:25Z" level=debug msg="channel.NewClient: waiting before trying to connect again..."
time="2020-02-11T19:46:27Z" level=debug msg="channel.NewClient: connect wait timeout (waited=1m8.002820802s)"
time="2020-02-11T19:46:27Z" level=error msg="channel.NewCommandClient: NewClient error = wait timeout"
time="2020-02-11T19:46:27Z" level=error msg="ipc.NewClient init error = wait timeout"
time="2020-02-11T19:46:27Z" level=fatal msg="docker-slim: failure" error="wait timeout" stack="goroutine 1 [running]:\nruntime/debug.Stack(0xc000145de8, 0x4c6a78, 0x0)\n\truntime/debug/stack.go:24 +0x9d\ngithub.com/docker-slim/docker-slim/pkg/util/errutil.FailOn(0xb6bea0, 0xc000037700)\n\tgithub.com/docker-slim/docker-slim@/pkg/util/errutil/errutil.go:14 +0x54\ngithub.com/docker-slim/docker-slim/internal/app/master/commands.OnBuild(0x9c4e01, 0xa92aae, 0x10, 0x1, 0xc000027c00, 0x2e, 0xa93194, 0x11, 0xa80001, 0xc00011f470, ...)\n\tgithub.com/docker-slim/docker-slim@/internal/app/master/commands/build.go:247 +0x1318\ngithub.com/docker-slim/docker-slim/internal/app/master.init.0.func6(0xc0000aa9a0, 0x0, 0xc0000b0db0)\n\tgithub.com/docker-slim/docker-slim@/internal/app/master/cli.go:775 +0x1709\ngithub.com/urfave/cli.HandleAction(0x9b03a0, 0xab7478, 0xc0000aa9a0, 0xc0000aa9a0, 0x0)\n\tgithub.com/urfave/cli@v1.22.1/app.go:523 +0xbe\ngithub.com/urfave/cli.Command.Run(0xa8978f, 0x5, 0x0, 0x0, 0xc000148530, 0x1, 0x1, 0xaab688, 0x3e, 0x0, ...)\n\tgithub.com/urfave/cli@v1.22.1/command.go:174 +0x51c\ngithub.com/urfave/cli.(*App).Run(0xc000158000, 0xc00001e0c0, 0xc, 0xc, 0x0, 0x0)\n\tgithub.com/urfave/cli@v1.22.1/app.go:276 +0x718\ngithub.com/docker-slim/docker-slim/internal/app/master.runCli()\n\tgithub.com/docker-slim/docker-slim@/internal/app/master/cli.go:1159 +0x55\ngithub.com/docker-slim/docker-slim/internal/app/master.Run()\n\tgithub.com/docker-slim/docker-slim@/internal/app/master/app.go:6 +0x25\nmain.main()\n\tgithub.com/docker-slim/docker-slim@/cmd/docker-slim/main.go:8 +0x20\n" version="linux|Transformer|1.26.1|2ec04e169b12a87c5286aa09ef44eac1cea2c7a1|2019-11-28_04:37:59PM"

My Dockerfile is pretty simple (a starter):

FROM centos:6.10
LABEL maintainer="Me"
ENV DOCKER_CONTENT_TRUST=1
RUN sed -i "/distroverpkg=centos-release/a override_install_langs=en_US.utf8\ntsflags=nodocs" /etc/yum.conf
RUN yum update -y
RUN yum install -y curl
RUN yum install -y openssl
RUN yum install -y ca-certificates
RUN update-ca-trust force-enable
RUN yum clean all

The command contains what I think is necessary: dist_linux/docker-slim --debug --verbose --in-container build centos-6.10 --tag centos-6.10-slim --http-probe=false --continue-after=1 --show-clogs --show-blogs

The docker-slim container is correctly launched:

CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                                                NAMES
16cbabfebfea        centos-6.10         "/opt/dockerslim/bin…"   About a minute ago   Up About a minute   0.0.0.0:32769->65501/tcp, 0.0.0.0:32768->65502/tcp   dockerslimk_49_20200211194518

And it seems the container waits for the command with no error:

 time="2020-02-11T19:45:18Z" level=debug msg="sensor: uid=0 euid=0"
time="2020-02-11T19:45:18Z" level=debug msg="sensor: sysinfo => system.SystemInfo{Sysname:\"Linux\", Nodename:\"16cbabfebfea\", Release:\"3.10.0-1062.9.1.el7.x86_64\", Version:\"#1 SMP Fri Dec 6 15:49:49 UTC 2019\", Machine:\"x86_64\", Domainname:\"(none)\", OsName:\"unknown\", OsBuild:\"\"}"
time="2020-02-11T19:45:18Z" level=debug msg="sensor: kernel flags => map[string]string(nil)"
time="2020-02-11T19:45:18Z" level=info msg="sensor: args => []string{\"/opt/dockerslim/bin/docker-slim-sensor\", \"-d\"}"
time="2020-02-11T19:45:18Z" level=debug msg="sensor: cwd => \"/\""
time="2020-02-11T19:45:18Z" level=debug msg="sensor: setting up channels..."
time="2020-02-11T19:45:18Z" level=debug msg="sensor: ipc.Server.Run()"
time="2020-02-11T19:45:18Z" level=debug msg="channel.Server.Start() - addr=0.0.0.0:65502 [time=1581450318970775521]"
time="2020-02-11T19:45:18Z" level=debug msg="channel.Server.Start() - addr=0.0.0.0:65501 [time=1581450318971080394]"
time="2020-02-11T19:45:18Z" level=info msg="sensor: waiting for commands..."
time="2020-02-11T19:45:18Z" level=debug msg="channel.Server.Start.loop()... [time=1581450318971246231]"
time="2020-02-11T19:45:18Z" level=debug msg="sensor: ipc.Server.Run - waiting for done signal..."
time="2020-02-11T19:45:18Z" level=debug msg="channel.Server.Start.loop()... [time=1581450318971557664]"
time="2020-02-11T19:45:18Z" level=debug msg="sensor: error collector - waiting for errors..."
time="2020-02-11T19:45:23Z" level=debug msg=.
time="2020-02-11T19:45:28Z" level=debug msg=.
time="2020-02-11T19:45:33Z" level=debug msg=.
time="2020-02-11T19:45:38Z" level=debug msg=.
time="2020-02-11T19:45:43Z" level=debug msg=.
time="2020-02-11T19:45:48Z" level=debug msg=.
time="2020-02-11T19:45:53Z" level=debug msg=.
time="2020-02-11T19:45:58Z" level=debug msg=.
time="2020-02-11T19:46:03Z" level=debug msg=.
time="2020-02-11T19:46:08Z" level=debug msg=.
time="2020-02-11T19:46:13Z" level=debug msg=.
time="2020-02-11T19:46:18Z" level=debug msg=.
time="2020-02-11T19:46:23Z" level=debug msg=.

Any idea please?
Appreciated

Start with internal/app/master/commands/build.go, especially the build path that creates the inspected container and connects to ports 65501 and 65502. Reproduce the timeout using the provided CentOS 6.10 image and in-container GitLab CI setup, then trace the IPC connection against the sensor logs. Done means the root cause is identified and the build no longer fails with this wait timeout, with a regression test if the existing test structure supports it.