mattermost / mattermost/mattermost-operator
Inconsistent server behavior with several replicas (k8s)
- Dominant language
- Go
- Stars
- 143
- Forks
- 90
- Avg merge
- 5d 23h
- Merged PRs (30d)
- 2
Description
#### Summary
Messages not delivered to client if there is several server replicas running.
#### Steps to reproduce
Server running using Mattermost Kubernetes Operator.
Version: 7.5.2
1. scale installation to 1 replica. Mattermost operates nicely
2. scale installation to 2 or more replicas. Missed messages
#### Expected behavior
Software operates the same no matter how much replicas run
#### Observed behavior (that appears unintentional)
On screenshots:
* right window - account who send messages (web version)
* left top - account who receie messages (Linux desktop client ver 5.2.2)
* left bottom - account who receie messages (web version)
1 server replica. All messages delivered:

2 serer replicas. Messages delivered to client randomly. In this example I use two clients authenticated to same account. While testing I see situations when both clients receive and "loss" same messages or they "see" different messages (as pictured on screenshot). If refresh all messages shown.

#### Possible fixes
This situation arise week ago, we run 2 replicas server before and didn't suffer from any troubles (or didn't notice).
About same time we have our Kubernetes cluster upgraded from v1.23.x to v1.24.8. We use DigitalOcean managed k8s. Ingress controller - Kong.
Maybe some sort of sticky session will help, didn't try yet
Contributor guide
Research direction
Start by reproducing message delivery with one versus multiple Mattermost server replicas under the Kubernetes Operator. Compare the Kubernetes 1.23 and 1.24 environments and inspect the Kong ingress configuration, including whether session handling differs. Done means messages are delivered consistently to all clients when multiple replicas run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- distributed-systems, infrastructure, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100