mattermost / mattermost/mattermost-operator

Inconsistent server behavior with several replicas (k8s)

Open
#331 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
143
Forks
90
Avg merge
5d 23h
Merged PRs (30d)
2

Description

#### Summary
Messages not delivered to client if there is several server replicas running.

#### Steps to reproduce
Server running using Mattermost Kubernetes Operator.
Version: 7.5.2

1. scale installation to 1 replica. Mattermost operates nicely
2. scale installation to 2 or more replicas. Missed messages

#### Expected behavior
Software operates the same no matter how much replicas run

#### Observed behavior (that appears unintentional)

On screenshots:
* right window - account who send messages (web version)
* left top - account who receie messages (Linux desktop client ver 5.2.2)
* left bottom - account who receie messages (web version)

1 server replica. All messages delivered:
![image](https://user-images.githubusercontent.com/5183334/209699212-9406c03d-372f-4157-a4ac-854d53ccea36.png)

2 serer replicas. Messages delivered to client randomly. In this example I use two clients authenticated to same account. While testing I see situations when both clients receive and "loss" same messages or they "see" different messages (as pictured on screenshot). If refresh all messages shown.
![image](https://user-images.githubusercontent.com/5183334/209698696-ef7ff4b3-fc34-42b2-98b6-1b0b61ba4444.png)

#### Possible fixes
This situation arise week ago, we run 2 replicas server before and didn't suffer from any troubles (or didn't notice).
About same time we have our Kubernetes cluster upgraded from v1.23.x to v1.24.8. We use DigitalOcean managed k8s. Ingress controller - Kong.
Maybe some sort of sticky session will help, didn't try yet

Contributor guide

Open the contributing guide

Research direction

Start by reproducing message delivery with one versus multiple Mattermost server replicas under the Kubernetes Operator. Compare the Kubernetes 1.23 and 1.24 environments and inspect the Kong ingress configuration, including whether session handling differs. Done means messages are delivered consistently to all clients when multiple replicas run.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
distributed-systems, infrastructure, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.