envoyproxy / envoyproxy/nighthawk

Consider generalizing the adaptive load "testing stage"

Open
#539 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
414
Forks
95
Avg merge
2d 9h
Merged PRs (30d)
11

Description

Today we have to deal with the adjusting stage and testing stage having separate durations, and the fact that the traffic template is a full `CommandLineOptions` with a duration field that is ignored.

One basic idea I had was to make `StepController::GetLatestCommandLineOptions()` take a duration input, so the `CommandLineOptions` is completely ready to send. But in some sense that just pushes the weird treatment of duration to another place.

I'm not sure that the idea of a dedicated testing stage really needs to be baked into the framework. During the design review we talked about exploring the input space, rather than just searching for a single target RPS. After such an exploration, the adaptive load framework would force you to do a testing stage you aren't interested in, and you would discard the result.

More generic would be to have the `StepController` decide everything, with duration being no different from anything else in `CommandLineOptions`. Some `StepController`s, like the basic one, could have a "search and then test" behavior where they run one longer benchmark at the end, but it would be the `StepController` deciding it. The main loop could just do whatever the `StepController` wants until it says it's time to exit, then not have any testing stage.

Concretely, this would mean:
- Deleting the durations from the `AdaptiveLoadSessionSpec` proto
- Adding duration settings to each `StepController`
- Deleting the testing stage code that follows the main loop
- Deleting the testing stage from the output proto

Duration is certainly an `InputVariableSetter` plugin we could trivially support, as suggested in https://github.com/envoyproxy/nighthawk/pull/535#pullrequestreview-488483737

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.