kubernetes-sigs / kubernetes-sigs/devops-bench
chaos: construct the fortio load argv from structured parameters instead of a model-composed command
- Dominant language
- Python
- Stars
- 29
- Forks
- 15
- Avg merge
- 6d 3h
- Merged PRs (30d)
- 15
Description
The `generate_load` fault exposes a free-form `run_command` tool to the chaos agent: the model composes a `fortio load` shell string that is tokenized and executed on the harness host (`run_chaos_command` in `devops_bench/chaos/faults/generate_load.py`). The executable, flags, and target URL are all model-chosen, so a misbehaving or prompt-injected model could run other binaries or direct load at hosts outside the eval.
Proposed change:
- Replace the `{command: string}` tool schema with structured parameters (`qps`, `duration`, `concurrency`).
- The tool handler constructs the fortio argv itself, pinned to the fault's target URL (the local port-forward tunnel or `LoadTarget.service_url`) — the model never supplies an executable or URL.
- Drop the string-parsing safeguards this makes unnecessary (`_LOAD_MARKER`, argv sniffing for `is_load`); the handler then knows definitively when a spike ran, simplifying the fail-closed accounting in `inject()`.
- Rewrite `build_system_instruction` so the model plans intensity and duration rather than composing shell strings.
- `ChaosAgent.tool_handler`'s contract changes from `(command: str, event)` to structured args — update the dispatch in `chaos/agent.py` and its tests.
Once the end-to-end pipeline can drive this fault against a live workload, re-validate that models reliably produce spikes through the structured tool.
/kind cleanup
/assign @pradeepvrd
Contributor guide
Assessment
This issue has not been assessed yet.