kubernetes-sigs / kubernetes-sigs/devops-bench

chaos: construct the fortio load argv from structured parameters instead of a model-composed command

Open
#33 0 comments 0 reactions 1 assignee Claimed by @pradeepvrd View on GitHub
kind/cleanup
Dominant language
Python
Stars
29
Forks
15
Avg merge
6d 3h
Merged PRs (30d)
15

Description

The `generate_load` fault exposes a free-form `run_command` tool to the chaos agent: the model composes a `fortio load` shell string that is tokenized and executed on the harness host (`run_chaos_command` in `devops_bench/chaos/faults/generate_load.py`). The executable, flags, and target URL are all model-chosen, so a misbehaving or prompt-injected model could run other binaries or direct load at hosts outside the eval.

Proposed change:
- Replace the `{command: string}` tool schema with structured parameters (`qps`, `duration`, `concurrency`).
- The tool handler constructs the fortio argv itself, pinned to the fault's target URL (the local port-forward tunnel or `LoadTarget.service_url`) — the model never supplies an executable or URL.
- Drop the string-parsing safeguards this makes unnecessary (`_LOAD_MARKER`, argv sniffing for `is_load`); the handler then knows definitively when a spike ran, simplifying the fail-closed accounting in `inject()`.
- Rewrite `build_system_instruction` so the model plans intensity and duration rather than composing shell strings.
- `ChaosAgent.tool_handler`'s contract changes from `(command: str, event)` to structured args — update the dispatch in `chaos/agent.py` and its tests.

Once the end-to-end pipeline can drive this fault against a live workload, re-validate that models reliably produce spikes through the structured tool.

/kind cleanup
/assign @pradeepvrd

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.