controller: error handling is hard

Let's use this as a tracking issue for things that need to be implemented or fixed in the controller relating to handling errors.

Background: The happy path of the controller is relatively simple, but many function calls can result in errors. Since this is the controller, what we do about each error is nuanced and complicated. For example, if we add a finalizer representing the test pod, then get an error when trying to create the test pod, then get an error when trying to remove the finalizer that represents the test pod, what should we do?

There will be an infinite number of possible scenarios, but let's track those that actually occur in practice and develop solutions for them using this tracking issue.