micronaut-projects / micronaut-projects/micronaut-http-benchmarks

Upstream subprocess-safe Hyperfoil standalone CLI execution

Open
#4 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Goal

Make the generated endpoint smoke benchmarks executable through stock Hyperfoil standalone CLI without maintaining project-specific CLI bootstrap or result-verifier classes.

This repository should ultimately be able to invoke a command similar to:

```bash
run.sh \
--standalone-root "$tmp/controller" \
--timeout 10m \
--fail-on-errors \
--json-result "$tmp/result.json" \
"$tmp/smoke.yaml"
```

## Motivation

Hyperfoil 0.29.3's one-shot `run.sh` is not a reliable subprocess interface for this use case:

- `LoadAndRun.main()` discards the result returned by `exec()`, so failures are not reliably reflected in the process exit status.
- The one-shot command does not expose the standalone controller root, despite `StartLocal` supporting one.
- `wait` has no timeout and can wait indefinitely.
- Runtime and validation errors are printed but do not necessarily fail the command.
- Machine-readable final results are only available indirectly.

Current Hyperfoil main already has a `BaseStandaloneCommand.startLocalCommand()` hook, making the root-directory change smaller than on 0.29.3.

## Proposed upstream work

- Change `LoadAndRun.main()` to propagate the result through `System.exit(...)`.
- Add `--standalone-root ` to the one-shot command and pass it to `start-local`.
- Add a bounded `--timeout ` option to `wait`/one-shot execution.
- Add `--fail-on-errors` so runtime, request-validation, and abnormal termination errors produce a non-zero exit status.
- Add optional `--json-result ` output, preferably reusing Hyperfoil's existing JSON export support.
- Preserve existing interactive behavior by making strict failure and timeout behavior opt-in where necessary.
- Add upstream CLI tests covering exit status, timeout, explicit root, cleanup, runtime errors, and JSON output.

## Repository integration

After the upstream changes are released:

- Update the pinned Hyperfoil version.
- Remove any local standalone CLI bootstrap and standalone result-verifier implementation.
- Invoke stock `run.sh` over the controller SSH connection.
- Keep the external process watchdog as a final infrastructure safeguard.
- Preserve temporary-file and mTLS keystore cleanup.

## Acceptance criteria

- A malformed benchmark exits non-zero.
- A runtime or response-validation failure exits non-zero.
- A timed-out run exits non-zero and cleans up the embedded controller.
- An explicit standalone root isolates persisted data for each invocation.
- A successful run exits zero and optionally writes stable machine-readable results.
- `micronaut-http-benchmarks` no longer packages custom Hyperfoil CLI or result-verifier classes.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with Hyperfoil's LoadAndRun.main(), BaseStandaloneCommand.startLocalCommand(), and the existing start-local/wait command paths. Trace how one-shot execution reports results, handles timeout and cleanup, and exports JSON, then inspect this repository's generated benchmark invocation and custom bootstrap/verifier classes. Done means upstream CLI tests cover exit status, timeout, cleanup, explicit roots, runtime errors, and JSON output, followed by the repository integration changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, shell
Domain
cli, testing, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.