docker-compose multirole sometimes stuck on boot

The issue is about docker-compose-multirole.yaml example in temporalio/docker-compose repo. I am posting this here because temporalio/docker-compose does not have an issue page.

Expected Behavior

When I run docker compose -f docker-compose-multirole.yaml up the whole multirole cluster is up and running normally.

Actual Behavior

When I run docker compose -f docker-compose-multirole.yaml up, temporal-history is sometimes stuck at "Waiting for Temporal server to start...", unable to reach the frontend service via nginx. The whole cluster does not seem to be in sync because I cannot connect to the UI service as well.

This does not always happen. So you need to try this multiple times. To me it feels like the chance is about 20%, especially when you stop docker-compose after running the service for a long time.

Restarting temporal-frontend or temporal-frontend2 sometimes make the cluster to the normal state, but not always.

Steps to Reproduce the Problem

1. git clone https://github.com/temporalio/docker-compose
2. cd docker-compose
3. docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
4. docker compose -f docker-compose-multirole.yaml up
5. If it works well, press control+C to stop the docker-compose project and wait for a moment. Then run the previous command again until you come across the problem.

Specifications

• Version: 1.22.4 (it happens in the previous versions too)
• Platform: macOS Sonoma 14.3.1 (newest version)
• Docker Desktop: 4.27.2 (newest version)

Here is the following log:

temporal-nginx          | 192.168.16.6 - - [27/Feb/2024:13:03:16 +0000] "POST /grpc.health.v1.Health/Check HTTP/2.0" 204 0 "-" "grpc-go/1.59.0"
temporal-nginx          | 2024/02/27 13:03:16 [error] 29#29: *73 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.16.6, server: , request: "POST /grpc.health.v1.Health/Check HTTP/2.0", upstream: "grpc://192.168.16.8:7237", host: "temporal-nginx:7233"
temporal-nginx          | 2024/02/27 13:03:16 [error] 29#29: *73 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.16.6, server: , request: "POST /grpc.health.v1.Health/Check HTTP/2.0", upstream: "grpc://192.168.16.9:7236", host: "temporal-nginx:7233"
temporal-history        | Error: unable to health check "temporal.api.workflowservice.v1.WorkflowService" service: unexpected HTTP status code received from server: 204 (No Content); malformed header: missing HTTP content-type
temporal-history        | ('export TEMPORAL_CLI_SHOW_STACKS=1' to see stack traces)
temporal-history        | Waiting for Temporal server to start...
temporal-nginx          | 2024/02/27 13:03:18 [error] 29#29: *76 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.16.6, server: , request: "POST /grpc.health.v1.Health/Check HTTP/2.0", upstream: "grpc://192.168.16.9:7236", host: "temporal-nginx:7233"
temporal-nginx          | 2024/02/27 13:03:18 [error] 29#29: *76 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.16.6, server: , request: "POST /grpc.health.v1.Health/Check HTTP/2.0", upstream: "grpc://192.168.16.8:7237", host: "temporal-nginx:7233"
temporal-nginx          | 192.168.16.6 - - [27/Feb/2024:13:03:18 +0000] "POST /grpc.health.v1.Health/Check HTTP/2.0" 204 0 "-" "grpc-go/1.59.0"
temporal-history        | Error: unable to health check "temporal.api.workflowservice.v1.WorkflowService" service: unexpected HTTP status code received from server: 204 (No Content); malformed header: missing HTTP content-type
temporal-history        | ('export TEMPORAL_CLI_SHOW_STACKS=1' to see stack traces)
temporal-history        | Waiting for Temporal server to start...
temporal-history        | Error: unable to health check "temporal.api.workflowservice.v1.WorkflowService" service: unexpected HTTP status code received from server: 204 (No Content); malformed header: missing HTTP content-type
temporal-history        | ('export TEMPORAL_CLI_SHOW_STACKS=1' to see stack traces)
temporal-nginx          | 2024/02/27 13:03:19 [error] 29#29: *79 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.16.6, server: , request: "POST /grpc.health.v1.Health/Check HTTP/2.0", upstream: "grpc://192.168.16.8:7237", host: "temporal-nginx:7233"
temporal-nginx          | 2024/02/27 13:03:19 [error] 29#29: *79 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.16.6, server: , request: "POST /grpc.health.v1.Health/Check HTTP/2.0", upstream: "grpc://192.168.16.9:7236", host: "temporal-nginx:7233"
temporal-nginx          | 192.168.16.6 - - [27/Feb/2024:13:03:19 +0000] "POST /grpc.health.v1.Health/Check HTTP/2.0" 204 0 "-" "grpc-go/1.59.0"
temporal-history        | Waiting for Temporal server to start...
temporal-history        | Error: unable to health check "temporal.api.workflowservice.v1.WorkflowService" service: unexpected HTTP status code received from server: 204 (No Content); malformed header: missing HTTP content-type
temporal-history        | ('export TEMPORAL_CLI_SHOW_STACKS=1' to see stack traces)
temporal-nginx          | 2024/02/27 13:03:20 [error] 29#29: *82 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.16.6, server: , request: "POST /grpc.health.v1.Health/Check HTTP/2.0", upstream: "grpc://192.168.16.9:7236", host: "temporal-nginx:7233"
temporal-nginx          | 2024/02/27 13:03:20 [error] 29#29: *82 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.16.6, server: , request: "POST /grpc.health.v1.Health/Check HTTP/2.0", upstream: "grpc://192.168.16.8:7237", host: "temporal-nginx:7233"
temporal-nginx          | 192.168.16.6 - - [27/Feb/2024:13:03:20 +0000] "POST /grpc.health.v1.Health/Check HTTP/2.0" 204 0 "-" "grpc-go/1.59.0"
temporal-history        | Waiting for Temporal server to start...

Setting TEMPORAL_CLI_SHOW_STACKS=true on temporal-history does not help much:

temporal-history        | Error: unable to health check "temporal.api.workflowservice.v1.WorkflowService" service: unexpected HTTP status code received from server: 204 (No Content); malformed header: missing HTTP content-type
temporal-history        | Stack trace:
temporal-history        | goroutine 1 [running]:
temporal-history        | runtime/debug.Stack()
temporal-history        |       /opt/hostedtoolcache/go/1.20.11/x64/src/runtime/debug/stack.go:24 +0x64
temporal-history        | runtime/debug.PrintStack()
temporal-history        |       /opt/hostedtoolcache/go/1.20.11/x64/src/runtime/debug/stack.go:16 +0x1c
temporal-history        | github.com/temporalio/cli/app.HandleError(0x40003df438?, {0x2b492a0, 0x40007481e0})
temporal-history        |       /home/runner/work/cli/cli/app/app.go:73 +0x134
temporal-history        | github.com/urfave/cli/v2.(*App).handleExitCoder(0x40009d91e0?, 0x400014de00?, {0x2b492a0?, 0x40007481e0?})
temporal-history        |       /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/app.go:452 +0x3c
temporal-history        | github.com/urfave/cli/v2.(*Command).Run(0x40009d91e0, 0x40003eb480, {0x4000b10200, 0x1, 0x1})
temporal-history        |       /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/command.go:276 +0x768
temporal-history        | github.com/urfave/cli/v2.(*Command).Run(0x406ef60, 0x40003eb340, {0x4000b01be0, 0x2, 0x2})
temporal-history        |       /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/command.go:267 +0x948
temporal-history        | github.com/urfave/cli/v2.(*Command).Run(0x406f900, 0x40003eb280, {0x40009c92f0, 0x3, 0x3})
temporal-history        |       /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/command.go:267 +0x948
temporal-history        | github.com/urfave/cli/v2.(*Command).Run(0x40009dab00, 0x40003eb140, {0x400004c0c0, 0x4, 0x4})
temporal-history        |       /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/command.go:267 +0x948
temporal-history        | github.com/urfave/cli/v2.(*App).RunContext(0x40008da780, {0x2b6a678?, 0x4000058068}, {0x400004c0c0, 0x4, 0x4})
temporal-history        |       /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/app.go:332 +0x568
temporal-history        | github.com/urfave/cli/v2.(*App).Run(0x60?, {0x400004c0c0?, 0x400007c768?, 0x49b54?})
temporal-history        |       /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/app.go:309 +0x40
temporal-history        | main.main()
temporal-history        |       /home/runner/work/cli/cli/cmd/temporal/main.go:14 +0x38
temporal-history        | Waiting for Temporal server to start...

Start with docker-compose-multirole.yaml in the referenced temporalio/docker-compose repository and reproduce the failure using docker compose up, including a stop and restart cycle. Read the temporal-history and temporal-nginx logs, focusing on the health-check responses and refused upstream connections. Done means the multirole cluster starts reliably after repeated restarts, with temporal-history progressing and the UI reachable.