argoproj / argoproj/notifications-engine

Slack groupingKey thread_ts not persisted - threading breaks after controller restart or leader change

Open
#462 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
334
Forks
217
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**

When using groupingKey in a Slack notification template, subsequent notifications for the same rollout are expected to be sent as replies in the same Slack thread. This works correctly within a single controller process lifetime, but threading breaks whenever the controller restarts or leadership changes in a multi-replica setup.

The root cause is that when a Slack message is sent with a groupingKey, the Slack API returns a thread_ts - the unique timestamp ID of the sent message. Currently, thread_ts is stored only in the memory of the controller pod that sent the original message. It is never persisted to Kubernetes (not in annotations, Secrets, or ConfigMaps).

As a result, when the controller pod restarts or leadership changes in a multi-replica deployment, the thread_ts is lost and the next notification creates a new top-level Slack message instead of replying in the existing thread.

**To Reproduce**

1. Deploy argo-rollouts with replicas: 2
2. Configure a Rollout with a Slack notification template using groupingKey
3. Trigger a rollout - observe the "Start Running" Slack message and thread
4. Let the rollout reach a Pause step - observe the Pause notification sent as a reply in the thread ✓
5. Restart the argo-rollouts controller pods (or wait for a natural leader change)
6. Restart the Rollout(e.g. via kubectl argo rollouts restart )
7. Observe the new Pause notification - it creates a new top-level Slack message instead of replying in the existing thread ✗

**Expected behavior**

All notifications for the same groupingKey are sent as replies in the same Slack thread, regardless of which controller pod processes them or how many times the controller restarts.

**The groupingKey is configured as:**

groupingKey: "...{{ .rollout.status.currentPodHash }}"
The intent is that all notifications for the same rollout revision (same pod hash) are grouped into a single Slack thread.
This works correctly within a single controller process lifetime but breaks as described.

**Failure conditions**

The threading breaks in any of the following cases:

1. Leader pod restarts - slackState is reset on startup regardless of whether the same pod wins the leader election again or not. The thread_ts is gone either way.
2. Leader changes without a restart - In a multi-replica setup, leader election can switch between pods due to a lease timeout or network partition. The new leader pod has its own separate slackState with an empty ThreadTSs map, with no knowledge of thread_ts values stored by the previous leader.

In all cases, the next notification creates a new top-level Slack message instead of a reply in the existing thread.

**Version**

v1.9.0

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the Slack notification flow and the slackState ThreadTSs map described in the issue, then inspect how controller state survives Kubernetes pod restarts and leader changes. Reproduce the two-replica scenario and verify that the thread_ts is persisted and reused so subsequent notifications remain replies in the same groupingKey thread.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.