failed build with gitaly not found

Up front: this is not easily reproducible, so I provide it simply as notice and reference for anyone with similar symptoms.

Up front number 2: rebuild the image and try again. That's what worked for me.

I routinely build a local image of sameersbn/docker-gitlab (I tend to keep a month behind, so it's easier to keep it local to stay up-to-date). So working from the same Dockerfile, I just update to the newer last-month version and build. On a recent build, however, it seemed to work fine but on starting it, the docker log complained something like

can't find program "/usr/local/bin/gitaly"

(Actual error text might be different, working off memory here.) The image would stay running, but gitlab would effectively be unusable (503 error).

There must have been a glitch during the build that meant that gitaly was not installed. I don't have the build logs, so I cannot find if/where the problem occurred.

After a different rabbit-hole of troubleshooting, I decided to rebuild the image with no other changes, and voilà! I verified that /usr/local/bin/gitaly existed where it did not previously. And the image now starts/runs without complaint.

Takeaway for the maintainers of this repo: somewhere in the installation script (not certain if it is in your stuff or gitlab's), the installation of required sub-components does not properly fail if any of the sub-parts' installations fail. I don't know how to troubleshoot this, but it's something to keep in mind. Since the install.sh script does include set -e, apparently part of the gitaly installation portion can fail/timeout without triggering the parent-bash retcode checks.

• is it possible that exec_as_git can mask a failure condition?
• could gitaly's Makefile execute and work incompletely without the retcode being passed back?

If I can find a way to reliably reproduce this bug, I'll add to this issue. For now, just realize that there is something non-deterministic about the build process that might be remedied by re-running the build.

Start with the Dockerfile and assets/build/install.sh, especially lines 108-119 and the exec_as_git calls used for gitaly installation. Try to reproduce the missing /usr/local/bin/gitaly condition with build logs available, then verify that an installation failure is surfaced rather than producing a seemingly usable image.