O & B self-verification workflow

Nobody has claimed this yet.

The purpose of this issue is to organize discussion around orchestrator (O) and broadcaster (B) behavior in a workflow where orchestrators self-verify results before returning them to broadcasters. We assume that the verification method relies on a machine learning classifier that has a false negative rate > 0% (i.e. it is possible that the classifier labels a valid transcode result as an invalid transcode result). We also assume that O is penalized if it fails verification when B provides source & result data as well as signatures to a third party verifier. The goal of a self-verification workflow is to allow O to check if a result would fail verification (i.e. due to a false negative) and then change its behavior accordingly in order to avoid penalization when doing honest work.

## Results without signatures?

A basic sketch of a self-verification workflow is as follows:

1. O receives source
2. O transcodes the source and produces a result
3. O self-verifies the result
4. If O passes self-verification, it returns the result to B
5. If O fails self-verification, it does X

Candidates for X include:

1. Return the result without a signature
2. Return a special `InvalidResultError` error without the result

It is worth thinking about B's possible behavior given these candidates for X i.e. is one option more helpful for B such that B can expect that behavior from O and if the expectation is not met B can consider O faulty.

Before we discuss option 1, it is worth establishing why O's signature over the result is important. The signature cryptographically links the result to O's ETH address which is associated with on-chain stake held by a contract. Thus, if B has O's signature, it can present an invalid result to a third party verifier to penalize O. Additionally, the signature authenticates the sender of the result such that B knows that the result was produced by the O it expects to receive results from (based on O's ETH address). We might not need to be as reliant on signatures for authentication of a result sender if B & O can use a TLS enabled connection with [certificates self-signed using ETH keys](https://github.com/livepeer/go-livepeer/issues/437).

The potential benefit of option 1 is that B could still use the result in its playlist. However, this is only a benefit if B has a way to determine whether to use the result or not. B shouldn't just outright use a result without a signature because B does not know that it can penalize O if the result is faulty. So, B would just be blindly inserting the result into its playlist without a recourse mechanism. B could also verify the result if it does not have a signature but if verification fails it would be left in the same position that it would be in if it didn't verify the result at all. At the moment, I'm skeptical of the utility of returning a result without a signature and option 2 might accomplish the same thing.

## Retrying segments with other Os?

Regardless of whether O returns a result without a signature or a special `InvalidResultError` error, if B is not going to use the result in its playlist, what will B do instead? One possibility is that B will just re-submit the same source segment to a different O similar to what B would do if it encountered any other error. My concern here is whether other Os will similarly fail self-verification for the same source segment and then not return a signed result to B. The question to answer regarding the ML classifier based verification method is: "if 1 O fails self-verification for a given input, is it likely that all other Os will also fail self-verification for that same input?". If this answer to this question is yes, then given a false negative rate of X%, then B would expect to not receive signed results from any Os X% of the time (in a live context a segment might be dropped after a timeout if no signed results are received within that period of time). If the answer to this question is no, then re-submitting the source segment might be a reasonable solution.

Start by reviewing the self-verification workflow and the alternatives for handling results without signatures. Analyze whether different orchestrators are likely to fail on the same source segment, then document a decided behavior for O and B, including retry handling and the conditions for penalization.

Written by the indexing model from the issue text.