GoogleContainerTools / GoogleContainerTools/skaffold
Using `kubectl rollout status` may lead to connection churn and pauses
- Dominant language
- Go
- Stars
- 15.9k
- Forks
- 1.7k
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 10
Description
@npintaux is seeing long pauses with some deployments of the [Cloud Code Go-Guestbook](https://github.com/GoogleCloudPlatform/cloud-code-samples/tree/v1/golang/go-guestbook). When running with `-v debug`, he sees output like:
```
DEBU[0101] Fetching events for pod "go-guestbook-frontend-7dbd9f6695-w858j" subtask=-1 task=DevLoop
DEBU[0101] Running command: [kubectl --context connectgateway_np-argolis-anthos_global_np-eks rollout status deployment go-guestbook-frontend --namespace default --watch=false] subtask=-1 task=Deploy
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
DEBU[0149] Command output: [deployment "go-guestbook-mongodb" successfully rolled out
] subtask=-1 task=Deploy
- deployment/go-guestbook-mongodb is ready. [2/3 deployment(s) still pending]
- deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
DEBU[0156] Command output: [deployment "go-guestbook-backend" successfully rolled out
] subtask=-1 task=Deploy
- deployment/go-guestbook-backend is ready. [1/3 deployment(s) still pending]
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
DEBU[0282] Command output: [deployment "go-guestbook-frontend" successfully rolled out
] subtask=-1 task=Deploy
- deployment/go-guestbook-frontend is ready.
Deployments stabilized in 3 minutes 15.939 seconds
```
If you match up the `Running command`s with the `Command output`s, you can see that the final `Command output` for `go-guesbook-frontend` at t=0282 corresponds to the `kubectl rollout status` at t=0101.
Nicolas reports that running `kubectl rollout status` from the command-line during this pause returns much faster. He's trying to isolate whether this is an issue with the remote cluster or whether it's connection-establishment cost.
Regardless, it would be better if our deployment status checks could talk to the cluster directly over a shared connection rather than repeatedly calling out to `kubectl rollout status`.
Contributor guide
Assessment
This issue has not been assessed yet.