Joystream / Joystream/joystream

Benchmarks should always panic on failure

Open
#4,361 6 comments 1 reaction 0 assignees View on GitHub
benchmarking post-mainnet runtime tech-debt
Dominant language
Rust
Stars
1.4k
Forks
116
PR merge metrics
No merged PRs in 30d

Description

### The issue
Recently I observed very unusual behavior when generating weights for `project-token` pallet on https://github.com/Joystream/joystream/pull/4341.

The `project-token` `transfer` benchmark would fail with `MembershipHandleAlreadyRegistered` error, but only if project token account bloat bond in genesis config was set to `> 0`. This error on the surface seemed like something completely unrelated to the change that was causing it and turned out very hard to debug.

As it turned out, the issue was actually caused by insufficient balance during `issue_token` call [here](https://github.com/Joystream/joystream/blob/carthage/runtime-modules/project-token/src/benchmarking.rs#L250):

```
transfer {
let (owner_member_id, owner_account) = create_owner::();
/* ... */
let token_id = issue_token::(TransferPolicyParams::Permissionless)?;
/* ... */
```

As you can see, the `issue_token` call doesn't panic, but instead returns an error in case of failure.

This turned out to be problematic because of how the `benchmark` command executes the benchmarking functions, specifically here: https://github.com/paritytech/substrate/blob/master/utils/frame/benchmarking-cli/src/pallet/command.rs#L300.

In case `--no-verify` flag is **not** provided, for each benchmarking step, first a `StateMachine` instance is created that runs the benchmark function along with the `verify` logic. The result of this execution is **ignored** (unless there is a `panic` invoked), however the state changes are preserved in a mutable `changes` object (they are normally cleaned up after successful execution, however if the benchmarking function returns `Err`, the cleanup step is never reached). Then [a second `StateMachine` instance is created](https://github.com/paritytech/substrate/blob/master/utils/frame/benchmarking-cli/src/pallet/command.rs#L328), executing the same logic, however reusing a runtime state from the previous instance (`&mut changes`). The result of this execution is no longer ignored, it is properly decoded, and the main process fails in case it is `Err` (however it has little significance at this point and can be very confusing. It also won't tell us at which line the error occured if it didn't cause `panic`)

This is why a mysterious `MembershipHandleAlreadyRegistered` was occuring. The first `StateMachine` execution failed with `InsufficientJoyBalance` during `issue_token` call. This caused the database/state cleanup function to not be reached, however the error itself was ignored. During the next `StateMachine` execution, the `owner` member already existed (as the state was preserved), so the execution immediately failed with `MembershipHandleAlreadyRegistered` on `create_owner`.

### The solution
Make sure all benchmarking functions **do not** return `Err` on failure, either through `call()?;` syntax or explicitly.
Benchmarking functions should always panic on `Err`, so either `assert_ok!(result)` or `result.unwrap();` should be used instead.
This will prevent difficult-to-debug issues like the one mentioned above from occuring.

┆Issue is synchronized with this [Asana task](https://app.asana.com/0/1201958687417145/1203134848946002) by [Unito](https://www.unito.io)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.