[stress-testing] V2 deployment model
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 115
Description
Basically, the requirements for what we deploy and how we deploy it have grown from what we did in v1.
Originally, we envisioned a single container image and multiple tests in a single region and (probably) one chaos-mesh policy for all the tests.
I think the idea and scenarios of what people want to test has grown in a different direction and now what we have is:
1. Multiple tests, with different chaos-mesh scenarios (ie, not all of the scenarios want to do network stress, and each scenario might have it’s own different chaos-mesh policy)
2. The same tests will need to run in different container images, due to libc or runtime support (different versions of node, different versions of the JVM).
3. The tests should be deployable into multiple regions.
Today we’ve got our values.yaml as a starting point for providing parameters to the tests. Here's an example of what it looks like today:
```yaml
# today's values.yaml format
scenarios:
- "constantDetach"
- "constantDetachmentSender"
```
We should be able to just upgrade this file a bit and make it handle our different requirements.
So let's talk scenarios and caveats:
## 1. 1:1 chaos-mesh scenarios
This one shouldn't be too bad. Today we assume 1 chaos-mesh policy to all scenarios, and we should allow you to pick and choose the association. So maybe:
```yaml
# values.yaml with chaos-mesh associations
scenarios:
"constantDetach":
chaos-mesh: yaml-file-name-of-chaos-resource
"someotherscenario":
chaos-mesh: yaml-file-name-of-chaos-resource2
```
And not specifying a chaos-mesh policy just means you opt-opt entirely.
## 2. Different container images
I think this one is pretty similar. Today we just have a single Dockerfile and we did provide some limited configuration in the `Chart.yaml`:
```yaml
annotations:
stressTest: 'true' # enable auto-discovery of this test via `find-all-stress-packages.ps1`
namespace: 'go'
dockerbuilddir: '../..'
dockerfile: './Dockerfile'
```
We can just move this into the `values.yaml` and perhaps just add in a `dockerfiles` array:
```yaml
dockerfiles:
- Dockerfile.node16
- Dockerfile.node18
scenarios:
```
I did consider having them just specify the base image but I think we're better off giving flexibility here to use the standard Dockerfile things in case they actually do need to do something OS/image specific.
3. The tests should be deployable into multiple regions.
This one is slightly complicated because it requires us to modify the .bicep template they use for stress test resources and the individual "already-helmized" templates.
Once again I think we should just configure this in `values.yaml` and let the script handle this. The default (perfectly fine) assumption si that the resources will go wherever the `resourceGroup().location` is, which works well for what we want.
```bicep
// stress-test-resources.bicep
var location = resourceGroup().location
```
And now our values.yaml looks like this:
```yaml
# values.yaml
dockerfiles:
- Dockerfile.node16
- Dockerfile.node18
scenarios:
regions:
- uswest
- southeastasia
```
We should make this configurable for now since not all offerings are available in all regions. The work on our side is to make sure that we have an appropriate k8s node, which @benbp's already done for us with southeastasia. I don't think we need to go hog-wild with regions here, but picking a few exemplars from the major areas would be a good idea. Right now I'm perfectly content to have US and Asia, as we already have.
Contributor guide
Research direction
Start by reading the current values.yaml and Chart.yaml configuration, the stress-test-resources.bicep template, and the find-all-stress-packages.ps1 discovery flow. Define how scenarios map to chaos-mesh policies, Dockerfiles, and regions, then verify the deployment templates and script support those values without breaking the existing defaults.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, dockerfile, helm, kubernetes, yaml
- Domain
- cloud, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100