helm / helm/chartmuseum

loadtesting: request rate for uploading charts and get index drops a lot with 0.16.0 release

Open
#711 8 comments 0 reactions 1 assignee Claimed by @scbizu View on GitHub
Dominant language
Go
Stars
3.8k
Forks
407
PR merge metrics
No merged PRs in 30d

Description

Version:
0.16.0 release

Problem

Using locust to have the loadtesting, the request rate for uploading charts and get index drops a lot.

Description

Previously

Previously chartmuseum release version v0.12.0 with pvc:
locust -f locustfile.py --no-web -L DEBUG -c 200 -r 50 -n 100000 --csv=result
We define posting new charts and get index in locustfile.py. The request rate for uploading charts could arrive 300 req/s and get index 30 req/s

Using 0.16.0
But when we use 0.16.0, the request rate drops to 5 req/s and 0.5 req/s and after the charts arrives 230 or so, the post requests begin to fail.

![image](https://github.com/helm/chartmuseum/assets/40451093/cce19a1c-4651-400d-8589-a8e04e395a7e)
![image](https://github.com/helm/chartmuseum/assets/40451093/b9d66386-eab8-4781-8a9d-ab55ae39d99c)

From the log, you can see the latency is very high, {"L":"INFO","T":"2023-08-11T10:48:11.323Z","M":"[12781] Request served","path":"/api/mychart/charts","comment":"","clientIP":"XXXXX","method":"POST","statusCode":201,"latency":"19m6.78816919s","reqID":"xxxxxxxxxxxx"}

The chartmuseum pod cpu is not high only 50%
![image](https://github.com/helm/chartmuseum/assets/40451093/1bcb426d-9730-4e5e-b4c7-ed9b6852a18e)

Using go tool pprof to get some data:
![image](https://github.com/helm/chartmuseum/assets/40451093/0a2f9908-1a04-46f8-834a-ca67aa8a4f7d)
![image](https://github.com/helm/chartmuseum/assets/40451093/25cdc4f3-33e5-4f12-828a-65d1d25c92ea)
![image](https://github.com/helm/chartmuseum/assets/40451093/62bff2b6-dd28-4580-a3b2-f618fedeee0c)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.