hashicorp / hashicorp/vault-benchmark

Panic decoding JSON reports with `review` command

Open
#206 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
107
Forks
28
PR merge metrics
No merged PRs in 30d

Description

I'm attempting to review a benchmark report that I earlier saved as a JSON file, but `vault-benchmark` panics when deserializing the file:

```
$ vault-benchmark -version
vault-benchmark v0.3.0

$ vault-benchmark review -results_file benchmark-10nvs.json
error reading report: json: cannot unmarshal string into Go value of type vegeta.Metrics
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x8 pc=0x104e83754]

goroutine 1 [running]:
github.com/hashicorp/vault-benchmark/benchmarktests.(*Reporter).ReportTerse(0x0, {0x10534e580, 0x140001a2008})
github.com/hashicorp/vault-benchmark/benchmarktests/reports.go:117 +0x64
github.com/hashicorp/vault-benchmark/command.(*ReviewCommand).Run(0x14000300540, {0x140001a6060, 0x2, 0x2})
github.com/hashicorp/vault-benchmark/command/review.go:109 +0x270
github.com/mitchellh/cli.(*CLI).Run(0x140000e0140)
github.com/mitchellh/cli@v1.1.5/cli.go:262 +0x4a8
github.com/hashicorp/vault-benchmark/command.RunCustom({0x140001a6050?, 0x10468dd1c?, 0x1058e86f8?}, 0x0?)
github.com/hashicorp/vault-benchmark/command/main.go:138 +0x730
github.com/hashicorp/vault-benchmark/command.Run(...)
github.com/hashicorp/vault-benchmark/command/main.go:41
main.main()
github.com/hashicorp/vault-benchmark/main.go:20 +0x54
```

The JSON report was created through something like this:

```
vault-benchmark run -config benchmark.hcl -report_mode json > benchmark.json
```

Contributor guide

Open the contributing guide

Research direction

Start in command/review.go around ReviewCommand.Run and benchmarktests/reports.go around Reporter.ReportTerse, using the reported stack trace as the entry point. Reproduce the failure with the shown run and review commands, then trace the JSON decoding error into the nil reporter path. Done means reviewing a generated JSON report no longer panics and its decoding failure is handled clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.