Use more granular soft-fail

Using soft-fail on buildkite is kind of hitting a nail with a sledgehammer. If one of our soft-fail jobs fails, it could fail for any reason possible: the (expected) reason why we switched to soft-fail, a method error, an undefined variable. I think that we should instead define a soft-fail macro, and use that in the precise location where we might accept failures (e.g., benchmark timings that might fluctuate from job to job).

I'm bringing this up because the bi-diagonal matrix operator GPU job is failing, but it's not failing in the way that we originally allowed it to fail.

There doesn't seem to be a `@soft_fail` macro, but making one shouldn't be hard. Perhaps it's even not a bad idea to put it into a separate repo, in case it's more widely useful.

Start by inspecting the Buildkite soft-fail configuration and the bi-diagonal matrix operator GPU job mentioned in the issue. Determine where soft-fail is currently applied and how the accepted failure condition can be isolated. Done means expected benchmark fluctuations are tolerated while unrelated failures still fail the job.

Written by the indexing model from the issue text.