GitHub Action fails due to Docker Compose Call Sequence

Summary

The CI GitHub Action occasionally fails when executing the following step: https://github.com/rstudio/rsconnect-python/blob/master/.github/workflows/main.yml#L156-L182

See https://github.com/rstudio/rsconnect-python/actions/runs/5478386182/jobs/9999333833

In some instances, this results in the following error:

docker-compose up -d
rsconnect-python_rsconnect_1 is up-to-date
docker-compose exec -T rsconnect bash < vetiver-testing/setup-rsconnect/add-users.sh
Error response from daemon: Container 9bdc42da9e0c00adaa7765fde5cb31926403c905f2cd590f640bcc427092031e is restarting, wait until the container is running
make: *** [Makefile:173: dev] Error 1
Error: Process completed with exit code 2.

Reasoning

During the execution of the make dev target, two consecutive commands to docker-compose are issued here: https://github.com/rstudio/rsconnect-python/blob/master/.github/workflows/main.yml#L156-L182

The second command, docker-compose exec, causes an asynchronous container restart. This is not mentioned in the documentation and no logs are produced. But the restart can be viewed by watching docker ps during the execution of make dev.

Due to the asynchronous nature of the issued restart on "fast" machines, the shell script vetiver-testing/setup-rsconnect/add-users.sh will execute before the restart is complete. In some instances, this is not observed since the initialization sequence is observed before the shell script executes.

When this occurs, the shell script will log errors related to the cracklib-runtime. This is due to the cracklib-runtime not having finished initializing before the chpasswd command being issued.

RCA

On my local machine, I can recreate this issue using the following steps:

1. Open a terminal window and run while true; do docker ps; echo; done | while IFS= read -r line; do printf '[%s] %s\n' "$(date '+%H:%M:%S')" "$line"; done > log.
2. Execute make dev
3. After make dev completes. Stop the first command.
4. Inspect the log file and search for "Less than a second ago".

Note that lines containing this message are issued in two distinct blocks. The first block is the initial startup via docker-compose up -d. The second block comes from docker-compose exec ....

Beginne mit .github/workflows/main.yml in den Zeilen 156-182, dem make dev-Target in Makefile und vetiver-testing/setup-rsconnect/add-users.sh. Führe make dev aus und beobachte dabei docker ps wie beschrieben. Untersuche anschließend den verknüpften Actions-Lauf und das aufgezeichnete Log. Erledigt ist die Aufgabe, wenn der CI-Schritt nicht mehr mit dem Neustart des Containers kollidiert und die mit cracklib-runtime zusammenhängenden Fehler nicht auftreten.