On first start Temporal should not accept connections until Default namespace is setup

Expected Behavior

When Temporal accepts a client connection it should be in a fully operable state.

Actual Behavior

On initial startup (using https://github.com/temporalio/temporal/blob/master/docker/docker-compose.yml), Temporal accepts connections from clients, but then fails when a worker attempts to Run() (via the sdk-go) with Namespace default does not exist.

The Temporal server log looks something like this:

1  : + echo 'waiting for cassandra to start up'
2  : + sleep 1
3  : + temporal-cassandra-tool --ep cassandra validate-health
4  : 2020/12/08 10:24:33 gocql: unable to dial control conn 172.28.0.2: dial tcp 172.28.0.2:9042: connect: connection refused
5  : 2020/12/08 10:24:33 unable to establish CQL session:gocql: unable to create session: control: unable to connect to initial hosts: dial tcp 172.28.0.2:9042: connect: connection refused
6  :
7  : ...
8  :
9  : + temporal-cassandra-tool --ep cassandra validate-health
10 : + echo 'cassandra started'
11 : + '[' '' '!=' true ']'
12 : + setup_schema
13 : + '[' cassandra == mysql ']'
14 : cassandra started
15 : + '[' cassandra == postgresql ']'
16 : + echo 'setup cassandra schema'
17 : + setup_cassandra_schema
18 : setup cassandra schema
19 :
20 : ...
21 :
22 : + temporal-cassandra-tool --ep cassandra validate-health
23 : + echo 'cassandra started'
24 : + '[' '' '!=' true ']'
25 : + setup_schema
26 : + '[' cassandra == mysql ']'
27 : cassandra started
28 : + '[' cassandra == postgresql ']'
29 : + echo 'setup cassandra schema'
30 : + setup_cassandra_schema
31 : setup cassandra schema
32 :
33 : ...
34 :
35 : {"level":"info","ts":"2020-12-08T10:24:49.675Z","msg":"Created gRPC listener","service":"history","address":"172.28.0.3:7234","logging-call-at":"rpc.go:134"}
36 : ...
37 : {"level":"info","ts":"2020-12-08T10:24:49.942Z","msg":"Created gRPC listener","service":"matching","address":"172.28.0.3:7235","logging-call-at":"rpc.go:134"}
38 : ...
39 : {"level":"info","ts":"2020-12-08T10:24:50.105Z","msg":"Created gRPC listener","service":"frontend","address":"172.28.0.3:7233","logging-call-at":"rpc.go:134"}
40 : ...
41 : {"level":"info","ts":"2020-12-08T10:24:50.263Z","msg":"Created gRPC listener","service":"worker","address":"172.28.0.3:7239","logging-call-at":"rpc.go:134"}
42 :
43 : ...
44 :
45 : + echo 'Registering default namespace: default'
46 : + tctl --ns default namespace describe
47 :
48 : Registering default namespace: default
49 : {"level":"info","ts":"2020-12-08T10:24:54.470Z","msg":"temporal-sys-history-scanner-workflow workflow successfully started","service":"worker","logging-call-at":"scanner.go:196"}
50 : {"level":"info","ts":"2020-12-08T10:24:54.475Z","msg":"Get dynamic config","name":"system.advancedVisibilityWritingMode","value":"off","default-value":"off","logging-call-at":"config.go:79"}
51 : {"level":"info","ts":"2020-12-08T10:24:54.475Z","msg":"Get dynamic config","name":"history.historyVisibilityOpenMaxQPS","value":"300","default-value":"300","logging-call-at":"config.go:79"}
52 : Error: Operation DescribeNamespace failed.
53 : Error Details: rpc error: code = NotFound desc = Namespace default does not exist.
54 : ('export TEMPORAL_CLI_SHOW_STACKS=1' to see stack traces)
55 : + echo 'Default namespace default not found. Creating...'
57 : + sleep 1
58 : Default namespace default not found. Creating...
59 : + tctl --ns default namespace register --rd 1 --desc 'Default namespace for Temporal Server'
60 : {"level":"info","ts":"2020-12-08T10:24:55.518Z","msg":"Get dynamic config","name":"history.enableAdminProtection","value":"false","default-value":"false","logging-call-at":"config.go:79"}
61 : {"level":"info","ts":"2020-12-08T10:24:55.519Z","msg":"Get dynamic config","name":"system.visibilityArchivalState","value":"enabled","default-value":"enabled","logging-call-at":"config.go:79"}
62 : {"level":"info","ts":"2020-12-08T10:24:55.552Z","msg":"Register namespace succeeded","service":"frontend","wf-namespace":"default","wf-namespace-id":"d13ba738-3b3c-4eb3-b60b-7dcbe75b275f","logging-call-at":"handler.go:278"}
63 : Namespace default successfully registered.
64 : + tctl --ns default namespace describe
65 : Name: default
66 : Id: d13ba738-3b3c-4eb3-b60b-7dcbe75b275f
67 : Description: Default namespace for Temporal Server
68 : OwnerEmail:
69 : NamespaceData: map[string]string(nil)
70 : State: Registered
71 : RetentionInDays: 24h0m0s
72 : ActiveClusterName: active
73 : Clusters: active
74 : HistoryArchivalState: Disabled
75 : VisibilityArchivalState: Disabled
76 : Bad binaries to reset:
77 : +-----------------+----------+------------+--------+
78 : | BINARY CHECKSUM | OPERATOR | START TIME | REASON |
79 : +-----------------+----------+------------+--------+
80 : +-----------------+----------+------------+--------+
81 : Default namespace registration complete.
82 : + echo 'Default namespace registration complete.'

From the client it looks like this:

• Lines 1-38:

  health check error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 172.28.0.3:7233: connect: connection refused
  

• Lines 39-80 (as can be seen from the server on lines 52/54 above):

  unable to start Temporal worker error="Namespace default does not exist."
  

• Lines 81+: All is good

Steps to Reproduce the Problem

1. Clone the https://github.com/temporalio/hello-world-project-template-go repo
2. Since the sample code exits on first failure, we can put in some retries... Replace worker/main.go with:

   func main() {
       // Create the client object just once per process
       var err error
       c, err = client.NewClient(client.Options{})
       if err != nil {
           // Retry up to 600 times, 1 second delay each time (5 minutes)
           const maxRetries = 600
           for i := 0; i < maxRetries; i++ {
               time.Sleep(time.Second)
               c, err = client.NewClient(topts)
               if err == nil {
                   break
               }
   
               // Only log after the first failed retry
               log.Warn("Temporal not available, retrying...")
           }
       }
   
       // If we still failed, exit
       c, err := client.NewClient(client.Options{})
       if err != nil {
           log.Fatalln("unable to create Temporal client", err)
       }
       defer c.Close()
       // This worker hosts both Worker and Activity functions
       w := worker.New(c, app.GreetingTaskQueue, worker.Options{})
       w.RegisterWorkflow(app.GreetingWorkflow)
       w.RegisterActivity(app.ComposeGreeting)
       // Start listening to the Task Queue
       err = w.Run(worker.InterruptCh())
       if err != nil {
           log.Fatalln("unable to start Worker", err)
       }
   }
   

3. go build worker
4. Get curl -O -J https://raw.githubusercontent.com/temporalio/temporal/master/docker/docker-compose.yml
5. docker-compose pull
6. At roughly the same time in two different shells:
   1. docker-compose up
   2. ./worker/worker

As described above, worker will eventually connect, and then fail when w.Run is called because the default namespace hasn't been created.

Additional Notes

While I could put a similar retry loop around the w.Run call, I would rather fail fast if the worker is misconfigured and using the wrong namespace or something like that. For connections a retry loop makes sense, and the Temporal client's internal grpc client will automatically reestablish the connection if it fails after the initial connect, but the initial connect seems to have no such logic.

Ideally, sdk-go would handle retries on both the client.NewClient call and the w.Run call, but there are no retry settings in ConnectionOptions or WorkerOptions. Happy to open a separate issue in that repo, but didn't want to spam the same team.

Seems like the easiest fix, for now, is to simply not accept connections until the initial setup is 100% complete.

Specifications

• Version: Temporal v1.3.2, sdk-go v1.2.0
• Platform: Ubuntu