argoproj / argoproj/notifications-engine

Error message in use with argo-rollouts

Open
#47 1 comment 3 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
334
Forks
217
PR merge metrics
No merged PRs in 30d

Description

I setup the notification-engine with the argo-rollouts deployment; everything looks good and messages are delivered to the slack channel as expected. However, I noticed that there are several error message each time some event is triggered:

```
INFO[2021-09-26T23:42:29-04:00] Start processing resource=default/rollouts-demo
INFO[2021-09-26T23:42:29-04:00] Reconciliation completed generation=3 namespace=default resourceVersion=2148175 rollout=rollouts-demo time_ms=15.550582
INFO[2021-09-26T23:42:29-04:00] Started syncing rollout generation=3 namespace=default resourceVersion=2148177 rollout=rollouts-demo
ERRO[2021-09-26T23:42:29-04:00] Failed to process: Object 'Kind' is missing in '{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"argoproj.io/v1alpha1\",\"kind\":\"Rollout\",\"metadata\":{\"annotations\":{\"notifications.argoproj.io/subscribe.on-rollout-completed.slack\":\"sdpl-test\",\"notifications.argoproj.io/subscribe.on-rollout-step-completed.slack\":\"sdpl-test\",\"notifications.argoproj.io/subscribe.on-rollout-updated.slack\":\"sdpl-test\"},\"name\":\"rollouts-demo\",\"namespace\":\"default\"},\"spec\":{\"replicas\":5,\"revisionHistoryLimit\":2,\"selector\":{\"matchLabels\":{\"app\":\"rollouts-demo\"}},\"strategy\":{\"canary\":{\"steps\":[{\"setWeight\":20},{\"analysis\":{\"args\":[{\"name\":\"duration\",\"value\":\"1s\"},{\"name\":\"exit-code\",\"value\":\"0\"}],\"templates\":[{\"templateName\":\"sleep-job\"}]}},{\"pause\":{}},{\"setWeight\":40},{\"pause\":{}},{\"setWeight\":60},{\"pause\":{}},{\"setWeight\":80},{\"pause\":{}}]}},\"template\":{\"metadata\":{\"labels\":{\"app\":\"rollouts-demo\"}},\"spec\":{\"containers\":[{\"image\":\"argoproj/rollouts-demo:blue\",\"name\":\"rollouts-demo\",\"ports\":[{\"containerPort\":8080,\"name\":\"http\",\"protocol\":\"TCP\"}],\"resources\":{\"requests\":{\"cpu\":\"5m\",\"memory\":\"32Mi\"}}}]}}}}\n","notifications.argoproj.io/subscribe.on-rollout-completed.slack":"sdpl-test","notifications.argoproj.io/subscribe.on-rollout-step-completed.slack":"sdpl-test","notifications.argoproj.io/subscribe.on-rollout-updated.slack":"sdpl-test","rollout.argoproj.io/revision":"3"},"creationTimestamp":"2021-09-27T02:28:28Z","generation":3,"managedFields":[{"apiVersion":"argoproj.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{},"f:notifications.argoproj.io/subscribe.on-rollout-completed.slack":{},"f:notifications.argoproj.io/subscribe.on-rollout-step-completed.slack":{},"f:notifications.argoproj.io/subscribe.on-rollout-updated.slack":{}}},"f:spec":{".":{},"f:replicas":{},"f:revisionHistoryLimit":{},"f:selector":{".":{},"f:matchLabels":{".":{},"f:app":{}}},"f:strategy":{".":{},"f:canary":{".":{},"f:steps":{}}},"f:template":{".":{},"f:metadata":{".":{},"f:labels":{".":{},"f:app":{}}},"f:spec":{".":{},"f:containers":{}}}}},"manager":"kubectl-client-side-apply","operation":"Update","time":"2021-09-27T02:28:28Z"},{"apiVersion":"argoproj.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}}},"manager":"kubectl-argo-rollouts-darwin-amd64","operation":"Update","time":"2021-09-27T03:37:42Z"},{"apiVersion":"argoproj.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:rollout.argoproj.io/revision":{}}},"f:status":{".":{},"f:HPAReplicas":{},"f:availableReplicas":{},"f:blueGreen":{},"f:canary":{},"f:conditions":{},"f:currentPodHash":{},"f:currentStepHash":{},"f:currentStepIndex":{},"f:message":{},"f:observedGeneration":{},"f:phase":{},"f:readyReplicas":{},"f:replicas":{},"f:selector":{},"f:stableRS":{},"f:updatedReplicas":{}}},"manager":"rollouts-controller","operation":"Update","time":"2021-09-27T03:42:29Z"}],"name":"rollouts-demo","namespace":"default","resourceVersion":"2148177","uid":"f7377772-b6a9-41d4-baaa-f25bd7d670a4"},"spec":{"replicas":5,"revisionHistoryLimit":2,"selector":{"matchLabels":{"app":"rollouts-demo"}},"strategy":{"canary":{"steps":[{"setWeight":20},{"analysis":{"args":[{"name":"duration","value":"1s"},{"name":"exit-code","value":"0"}],"templates":[{"templateName":"sleep-job"}]}},{"pause":{}},{"setWeight":40},{"pause":{}},{"setWeight":60},{"pause":{}},{"setWeight":80},{"pause":{}}]}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"rollouts-demo"}},"spec":{"containers":[{"image":"argoproj/rollouts-demo:red","name":"rollouts-demo","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"resources":{"requests":{"cpu":"5m","memory":"32Mi"}}}]}}},"status":{"HPAReplicas":5,"availableReplicas":4,"blueGreen":{},"canary":{},"conditions":[{"lastTransitionTime":"2021-09-27T03:37:42Z","lastUpdateTime":"2021-09-27T03:37:42Z","message":"Rollout is paused","reason":"RolloutPaused","status":"False","type":"Paused"},{"lastTransitionTime":"2021-09-27T03:42:28Z","lastUpdateTime":"2021-09-27T03:42:28Z","message":"RolloutCompleted","reason":"RolloutCompleted","status":"False","type":"Completed"},{"lastTransitionTime":"2021-09-27T03:42:29Z","lastUpdateTime":"2021-09-27T03:42:29Z","message":"Rollout does not have minimum availability","reason":"AvailableReason","status":"False","type":"Available"},{"lastTransitionTime":"2021-09-27T03:37:42Z","lastUpdateTime":"2021-09-27T03:42:29Z","message":"ReplicaSet \"rollouts-demo-6b5cbf7465\" is progressing.","reason":"ReplicaSetUpdated","status":"True","type":"Progressing"}],"currentPodHash":"6b5cbf7465","currentStepHash":"89b96d8b9","currentStepIndex":0,"message":"more replicas need to be updated","observedGeneration":"3","phase":"Progressing","readyReplicas":4,"replicas":5,"selector":"app=rollouts-demo","stableRS":"75f756d67b","updatedReplicas":1}}' resource=default/rollouts-demo
```

The above is the argo-rollouts controller's log, where I think the error message of the last line comes from the notification-engine. It could be

https://github.com/argoproj/notifications-engine/blob/3203039170aff46026e8c7fe1d82e62b8f330997/pkg/controller/controller.go#L229-L231

Although not harmful to the rollouts CR, the error messages are annoying. Do I miss anything in the configuration step? Thanks.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with pkg/controller/controller.go at lines 229-231, which the report identifies as the suspected error path, and reproduce a notification event through an argo-rollouts deployment. Compare the logged object and the controller's processing expectations; done means Slack notifications still arrive without the repeated “Object 'Kind' is missing” errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.