googleforgames / googleforgames/open-match
Open Match Query service pod gets OOMKilled
@ashutosji is already working on this.
Since Mar 7, 2023.
- Dominant language
- Go
- Stars
- 3.4k
- Forks
- 358
- PR merge metrics
- No merged PRs in 30d
Description
**What happened**:
This relates to https://github.com/googleforgames/open-match/issues/1390.
When we run our load test based on Open Match, we may generate huge amount of requests to Open Match Query service in a short period of time. We found that lots of Open Match Query service pods got `OOMKilled` like below:
```
kepler-loadtest-open-match-query-cd55b976-455ld 0/1 OOMKilled 1 3m10s
kepler-loadtest-open-match-query-cd55b976-bmgjw 0/1 Running 1 3m10s
kepler-loadtest-open-match-query-cd55b976-c5jch 1/1 Running 2 26m
kepler-loadtest-open-match-query-cd55b976-pbj5x 0/1 CrashLoopBackOff 1 2m55s
kepler-loadtest-open-match-query-cd55b976-vhdxw 1/1 Running 1 3m10s
kepler-loadtest-open-match-query-cd55b976-vz5x5 0/1 CrashLoopBackOff 3 26m
kepler-loadtest-open-match-query-cd55b976-w4kqk 0/1 OOMKilled 2 26m
kepler-loadtest-open-match-query-cd55b976-xrnkn 0/1 Running 1 2m55s
```
**What you expected to happen**:
We expect to see Open Match Query service handle high workload more smoothly instead of get `OOMKilled`. e.g. It can return the result slower or return an error indicating that it's overloaded.
**How to reproduce it (as minimally and precisely as possible)**:
Only have 3 pods of Open Match Query service running, then generate lots of query requests in a short period of time and each query should return several hundred of tickets. It seems that HPA is involved but it can not start enough pods quick enough, so existing pods have very high CPU and memory usage. And eventually they get OOMKilled.
**Anything else we need to know?**:
**Output of `kubectl version`**:
```
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-08T17:38:50Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.8-eks-96780e", GitCommit:"96780e1b30acbf0a52c38b6030d7853e575bcdf3", GitTreeState:"clean", BuildDate:"2021-03-10T21:32:29Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
```
**Cloud Provider/Platform (AKS, GKE, Minikube etc.)**:
EKS
**Open Match Release Version**:
1.1.0
**Install Method(yaml/helm):**:
helm
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.