bottlerocket-os / bottlerocket-os/bottlerocket-test-system
Agents: Support writing a test in bash
- Dominant language
- Rust
- Stars
- 18
- Forks
- 27
- PR merge metrics
- No merged PRs in 30d
Description
# Running a custom test requires building a docker container #
**Problem:** If a TestSys user wants to test features A, they will need to create a container containing the test they want to run.
**Solution:** Provide 2-3 new containers that are designed to integrate with bash scripts
## Bash Agent ##
The bash agent will accept an encoded bash script and run it. There are a few possible ways this can be expanded.
* Accept a set of scripts and run each in the order they were supplied
* Each script would be considered a test
* Users could specify on fail behavior (keep going or stop)
## SSM Bash Agent ##
The bash agent will accept an encoded bash script and run it over ssm to the provided instances. There are a few possible ways this can be expanded.
* Accept a set of scripts and run each in the order they were supplied
* Each script would be considered a test
* Users could specify on fail behavior (keep going or stop)
## Workload Bash Test ##
The bash agent will accept an encoded bash script and run it as a workload test. There are a few possible ways this can be expanded.
* Accept a set of scripts and run each in the order they were supplied
* Each script would be considered a test
* Users could specify on fail behavior (keep going or stop)
## Do we need all 3? ##
We will want at least 2 of the provided agents. The workload bash test would be useful for prototyping workload test containers like the Nvidia smoke tests and also testing bottlerocket changes that effect workloads.
The SSM Bash agent will be useful for applying setting changes via api client. This is a slight modification of the OS Test Agent.
The Bash agent will be useful for other testing workflows such as applying manifests to a cluster etc. This agent is not nearly as important as the other 2 and can be left out for now.
## SSM Bash Agent- ##
We can adapt the design from the original [OS test agent](#426) input so that instead of taking in a list of ssm docs it will take in
```
enum OsTest{
SsmDoc(String),
EncodedBash(String),
}
struct OsTestConfig{
/// The instances that should be targeted by this agent
instance_ids: Vec,
/// The tests that this agent will run
tests: Vec,
/// Variables that should be set in the environment before each test is run
env: BTreeMap
}
```
The applied yaml would look like
```
configuration:
tests:
- ssmDoc:
- ssmDoc:
- encodedBash:
env:
ENV_VAR_1: "an env variable"
ENV_VAR_2: "another env variable"
onFail: continue
instanceIds: ${ec2provider.instanceIds}
```
By accepting env variables as well, a user has full control over their testing. Specifying an OnFail behavior will tell the agent how to respond to a failure. For each test, an exit 0 will distinguish a successful test run.
## Workload Bash Agent- ##
Currently the workload agent accepts a set of WorkloadTest
```
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct WorkloadTest {
pub name: String,
pub image: String,
#[serde(default)]
pub gpu: bool,
}
#[derive(Debug, Clone, Default, Serialize, Deserialize, Configuration, Builder)]
#[serde(rename_all = "camelCase")]
#[crd("Test")]
pub struct WorkloadConfig {
pub kubeconfig_base64: String,
pub tests: Vec,
pub assume_role: Option,
}
```
We would change this to also accept env variables
```
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct WorkloadTest {
pub name: String,
pub image: String,
#[serde(default)]
pub gpu: bool,
pub env: BTreeMap,
}
```
The custom workload agent would take the value of ENCODED_BASH and base64 decode it to a bash file and run it.
## How much work is it going to take? ##
The SSM agent should be pretty straight forward, workload bash agent will be especially easy to implement. The complication of these agents will be the use interaction with cargo make test.
## `cargo make test` with the Workload Bash Agent ##
Currently, users can run a workload test by adding all of the workloads to the workload map
```
workloads = { = , = }
```
Users will continue to list their workloads that way. Before creating the WorkloadTest for each entry, if the test container uri is a file within the TestSys tests dir, that file will be encoded and added as an env variable and the bash workload agent will be used.
```
workloads = { myBashTest = my_test.sh, myOtherTest = my_test }
tests dir:
tests
|- shared
| |- my_test.sh
| |- workloads
| |- my_test.sh
```
The snippet above shows accepted file locations and acceptable ways of describing the path (with or without a file extension).
## `cargo make test` with the SSM Bash Agent ##
Currently, users can run a workload test by adding all of the workloads to the workload map
```
workloads = { = , = }
```
We will define ssm docs/os tests in a similar way.
```
os-tests = [my-ssm-doc, my-other-ssm-doc, my_bash_test.sh, my_other_bash_test]
```
Users will continue to list their SSM docs/bash scripts that way. Before creating the SSMTest for each entry, if the ssm doc uri is a file within the TestSys tests dir, that file will be encoded and added as a OsTest::EncodedBash() variant.
```
os-tests = { my_bash_test.sh, my_other_bash_test }
tests dir:
tests
|- shared
| |- my_bash_test.sh
| |- os-tests
| |- my_other_bash_test.sh
```
The snippet above shows accepted file locations and acceptable ways of describing the path (with or without a file extension).
Additionally, we will add a new dev configuration value called env.
```
[aws.dev]
env = { MY_ENV_VAR = "var1", MY_OTHER_ENV_VAR = "var2" }
```
All values from the env table will be added to the environment for all os-tests and workload tests.
Contributor guide
Research direction
The issue names no source files; start by reviewing the original OS test agent in issue #426, the workload agent configuration, and the cargo make test flows described here. Determine which two agents are in scope before implementation. Done should include the selected bash-based agents, encoded script handling, environment variables, failure behavior, and the documented test-directory integration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, bash, docker, rust
- Domain
- cloud, devops, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100