apache / apache/openwhisk

Setting up a new environment for performance tests

Open
#5,190 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
6.8k
Forks
1.2k
Avg merge
2d 14h
Merged PRs (30d)
2

Description

## Environment details:

* Travis VM

## Steps to reproduce the issue:

Some of the performance tests are being failed.
We need to migrate the environment for performance tests to the one with more resources.
We can consider introducing GHA with a VM provided by ASF.

## Provide the expected results and outputs:

There should be no activations with the "non-2xx,3xx" response.

## Provide the actual results and outputs:

```
The command "TERM=dumb ./tests/performance/wrk_tests/throughput.sh "https://172.17.0.1:10001" "$(cat ansible/files/auth.guest)" ./tests/performance/preparation/actions/noop.js 100 110 2 2m" exited with 0.
136.15s$ TERM=dumb ./tests/performance/wrk_tests/throughput.sh "https://172.17.0.1:10001" "$(cat ansible/files/auth.guest)" ./tests/performance/preparation/actions/async.js 100 110 2 2m
Creating action async_110
{"annotations":[{"key":"provide-api-key","value":false},{"key":"exec","value":"nodejs:14"}],"exec":{"kind":"nodejs:14","code":"/*\n * Licensed to the Apache Software Foundation (ASF) under one or more\n * contributor license agreements. See the NOTICE file distributed with\n * this work for additional information regarding copyright ownership.\n * The ASF licenses this file to You under the Apache License, Version 2.0\n * (the \"License\"); you may not use this file except in compliance with\n * the License. You may obtain a copy of the License at\n *\n * http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\nfunction main() {\n return new Promise(function (resolve, reject) {\n setTimeout(function () {\n resolve({done: true});\n }, 175);\n })\n}","binary":false},"limits":{"concurrency":110,"logs":10,"memory":256,"timeout":60000},"name":"async_110","namespace":"guest","parameters":[],"publish":false,"updated":1641263244008,"version":"0.0.1"}Running async_110 once to assert an intact system
{"activationId":"0e88ae3ed2fd4fb788ae3ed2fd7fb71d","annotations":[{"key":"path","value":"guest/async_110"},{"key":"waitTime","value":710},{"key":"kind","value":"nodejs:14"},{"key":"timeout","value":false},{"key":"limits","value":{"concurrency":110,"logs":10,"memory":256,"timeout":60000}},{"key":"initTime","value":388}],"duration":575,"end":1641263245366,"logs":[],"name":"async_110","namespace":"guest","publish":false,"response":{"result":{"done":true},"size":13,"status":"success","success":true},"start":1641263244791,"subject":"guest","version":"0.0.1"}Running 2m test @ https://172.17.0.1:10001/api/v1/namespaces/_/actions/async_110?blocking=true
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 47.17ms 26.73ms 258.88ms 69.63%
Req/Sec 0.88k 331.19 1.41k 82.12%
Latency Distribution
50% 43.86ms
75% 62.47ms
90% 81.93ms
99% 126.47ms
112787 requests in 2.00m, 55.47MB read
Socket errors: connect 0, read 0, write 0, timeout 434
Non-2xx or 3xx responses: 112353
```

## Additional information you deem important:
* Even with some failed activations, wrk tests themselves are not failed.

Contributor guide

Open the contributing guide

Research direction

Start with tests/performance/wrk_tests/throughput.sh and the Travis configuration, then review the referenced ansible/files/auth.guest and performance preparation actions. Reproduce the async.js throughput run and determine how a higher-resource VM or GitHub Actions environment should be configured. Done means the performance tests complete without non-2xx or 3xx responses and failures are reported correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
ansible, bash, github-actions, nodejs
Domain
ci-cd, infrastructure, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.