googleforgames / googleforgames/open-match

Open Match Query service pod gets OOMKilled

Open
#1,391 0 comments 0 reactions 1 assignee View on GitHub

@ashutosji is already working on this.

Since Mar 7, 2023.

kind/bug
Dominant language
Go
Stars
3.4k
Forks
358
PR merge metrics
No merged PRs in 30d

Description

**What happened**:

This relates to https://github.com/googleforgames/open-match/issues/1390.

When we run our load test based on Open Match, we may generate huge amount of requests to Open Match Query service in a short period of time. We found that lots of Open Match Query service pods got `OOMKilled` like below:

```
kepler-loadtest-open-match-query-cd55b976-455ld 0/1 OOMKilled 1 3m10s
kepler-loadtest-open-match-query-cd55b976-bmgjw 0/1 Running 1 3m10s
kepler-loadtest-open-match-query-cd55b976-c5jch 1/1 Running 2 26m
kepler-loadtest-open-match-query-cd55b976-pbj5x 0/1 CrashLoopBackOff 1 2m55s
kepler-loadtest-open-match-query-cd55b976-vhdxw 1/1 Running 1 3m10s
kepler-loadtest-open-match-query-cd55b976-vz5x5 0/1 CrashLoopBackOff 3 26m
kepler-loadtest-open-match-query-cd55b976-w4kqk 0/1 OOMKilled 2 26m
kepler-loadtest-open-match-query-cd55b976-xrnkn 0/1 Running 1 2m55s
```

**What you expected to happen**:

We expect to see Open Match Query service handle high workload more smoothly instead of get `OOMKilled`. e.g. It can return the result slower or return an error indicating that it's overloaded.

**How to reproduce it (as minimally and precisely as possible)**:

Only have 3 pods of Open Match Query service running, then generate lots of query requests in a short period of time and each query should return several hundred of tickets. It seems that HPA is involved but it can not start enough pods quick enough, so existing pods have very high CPU and memory usage. And eventually they get OOMKilled.

**Anything else we need to know?**:

**Output of `kubectl version`**:

```
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-08T17:38:50Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.8-eks-96780e", GitCommit:"96780e1b30acbf0a52c38b6030d7853e575bcdf3", GitTreeState:"clean", BuildDate:"2021-03-10T21:32:29Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
```

**Cloud Provider/Platform (AKS, GKE, Minikube etc.)**:

EKS

**Open Match Release Version**:

1.1.0

**Install Method(yaml/helm):**:

helm

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.