Soak Tests but for Configs We Can't Share

Today in Vector we have a notion of a 'soak test'. The soak test is an all-up, integrated benchmark for vector. It runs Vector with known configurations, against stable load generation in a repeatable environment and measures the throughput from the load generation side, comparing two SHAs in the process in a statistically significant way. We use this to control regressions and detect if optimizations actually have an impact when Vector is fully assembled and the most susceptible to Amdahl's law. The underlying mechanism is simplistic: minikube, some quiet EC2 machines, a bit of terraform, custom load generation, analysis scripts and a Github Action workflow.

At a high level we have a need for both performance and reliability tests in the Vector project. They are:

a. short duration performance soak tests
b. long duration performance soak tests
c. long duration memory reliability soak tests
d. short/long private customer configuration replication soak tests
e. statistically viable comparisons with competitor setups in a soak testing environment

The soak tests today cover point 'a' well. Covering 'b' is a matter of extending the runtime of the existing soaks. Covering 'c' is a matter of capture memory data from the vector pod in-kube. Point 'd' is not approachable with our current soak notion, in that we require configurations to be public and checked into the vector repository. Theoretically point 'e' is approachable with our current setup, though no work has been done to achieve this.

The current entry points are soaks/, soaks/bin/analyze_experiment, and .github/workflows/soak.yml; the infrastructure also uses minikube and Terraform. Start by tracing how configurations are supplied to the existing soak workflow and how experiments are analyzed. Done should be a repeatable soak path for private customer configurations while preserving the existing comparison and measurement behavior.