GoogleContainerTools / GoogleContainerTools/skaffold

Mysterious Warn logs in dev loop when container is terminated and watching for changes

Open
#5,565 0 comments 0 reactions 0 assignees View on GitHub
area/status-check bugbash/q4-2022 help wanted kind/friction priority/p2
Dominant language
Go
Stars
15.9k
Forks
1.7k
Avg merge
3d 9h
Merged PRs (30d)
10

Description

I am testing a scenario where users have a deployed container fails before status check phase is complete.

As expected Skaffold
1) reports the container is terminated with exit code.
2) Tries fetching logs which arent there.
3) quits the status check loop and watches for changes

Unexpected:
1) I see the following warn messages in the watching phase `WARN[0046] exit status 1 `
2) `WARN[0046] Skipping deploy due to error: 1/2 deployment(s) failed ` is incorrect because the deploy already happened and we don't clean up.

```
Waiting for deployments to stabilize...
- deployment/leeroy-web is ready. [1/2 deployment(s) still pending]
WARN[0045] exit status 1
- deployment/leeroy-app: container leeroy-app terminated with exit code 2
- pod/leeroy-app-794c7c586d-2qrh2: container leeroy-app terminated with exit code 2
> Error retrieving logs for pod leeroy-app-794c7c586d-2qrh2. Try `kubectl logs leeroy-app-794c7c586d-2qrh2 -n default -c leeroy-app`
- deployment/leeroy-app failed. Error: container leeroy-app terminated with exit code 2.
WARN[0046] Skipping deploy due to error: 1/2 deployment(s) failed
Watching for changes...
WARN[0046] exit status 1
WARN[0060] exit status 1
WARN[0090] exit status 1
````

How to reproduce
1) Add a lag in status check so the container has time to crash before status check ends.
```
+++ b/pkg/skaffold/deploy/status/status_check.go
@@ -97,6 +97,7 @@ func NewStatusChecker(cfg Config, labeller *label.DefaultLabeller) Checker {
// Run runs the status checks on deployments and pods deployed in current skaffold dev iteration.
func (s statusChecker) Check(ctx context.Context, out io.Writer) error {
event.StatusCheckEventStarted()
+ time.Sleep(1*time.Second)
errCode, err := s.statusCheck(ctx, out)
event.StatusCheckEventEnded(errCode, err)
return err
```
3) LOCAL=true make
4) run `skaffold dev`

5) After the first deploy, add a NPE to mimic container crash
```
--- a/examples/microservices/leeroy-app/app.go
+++ b/examples/microservices/leeroy-app/app.go
@@ -12,6 +12,7 @@ func handler(w http.ResponseWriter, r *http.Request) {

func main() {
log.Print("leeroy app server ready")
+ fmt.Fprintf(nil, "leeroooooy app!!\n")
http.HandleFunc("/", handler)
http.ListenAndServe(":50051", nil)
}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.