GoogleContainerTools / GoogleContainerTools/skaffold

Using `kubectl rollout status` may lead to connection churn and pauses

Open
#7,294 0 comments 0 reactions 0 assignees View on GitHub
area/status-check kind/friction source/partnerships
Dominant language
Go
Stars
15.9k
Forks
1.7k
Avg merge
3d 9h
Merged PRs (30d)
10

Description

@npintaux is seeing long pauses with some deployments of the [Cloud Code Go-Guestbook](https://github.com/GoogleCloudPlatform/cloud-code-samples/tree/v1/golang/go-guestbook). When running with `-v debug`, he sees output like:
```
DEBU[0101] Fetching events for pod "go-guestbook-frontend-7dbd9f6695-w858j" subtask=-1 task=DevLoop
DEBU[0101] Running command: [kubectl --context connectgateway_np-argolis-anthos_global_np-eks rollout status deployment go-guestbook-frontend --namespace default --watch=false] subtask=-1 task=Deploy
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
DEBU[0149] Command output: [deployment "go-guestbook-mongodb" successfully rolled out
] subtask=-1 task=Deploy
- deployment/go-guestbook-mongodb is ready. [2/3 deployment(s) still pending]
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
DEBU[0156] Command output: [deployment "go-guestbook-backend" successfully rolled out
] subtask=-1 task=Deploy
- deployment/go-guestbook-backend is ready. [1/3 deployment(s) still pending]
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
DEBU[0282] Command output: [deployment "go-guestbook-frontend" successfully rolled out
] subtask=-1 task=Deploy
- deployment/go-guestbook-frontend is ready.
Deployments stabilized in 3 minutes 15.939 seconds
```

If you match up the `Running command`s with the `Command output`s, you can see that the final `Command output` for `go-guesbook-frontend` at t=0282 corresponds to the `kubectl rollout status` at t=0101.

Nicolas reports that running `kubectl rollout status` from the command-line during this pause returns much faster. He's trying to isolate whether this is an issue with the remote cluster or whether it's connection-establishment cost.

Regardless, it would be better if our deployment status checks could talk to the cluster directly over a shared connection rather than repeatedly calling out to `kubectl rollout status`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.