Should the TUF update workflow eventually succeed even if non-volatile storage writes always fails?

In https://github.com/heartsucker/rust-tuf/pull/304, I'm extending rust-tuf to fail an update if writing to non-volatile storage fails. However, in https://github.com/heartsucker/rust-tuf/pull/304/files#r512292123, @wellsie1116 noticed an odd consequence to the update workflow, where even though the workflow fails on each storage, we're still updating our TUF trust database for each update attempt. So:

• update attempt 1:
  • we trust 1.root.json
  • 5.1, we fetch a new root metadata 2.root.json, and set it as our trusted root.
  • we try and fail to write 1.root.json to non-volatile storage, and so we fail the update
• update attempt 2:
  • 5.1, we try to fetch 3.root.json, but it doesn't exist
  • 5.2, we fetch timestamp.json, and set it as our trusted timestamp.
  • we try and fail to write timestamp.json to non-volatile storage, and so we fail the update.

And etc. I found this to be quite surprising, but I'm not sure if this is an actual problem. It sort of makes me wonder if it's really worthwhile failing a metadata update if we can't persist the metadata, and instead trying to signal to the user "we successfully updated the TUF metadata, but we couldn't persist the metadata because of ...". What do you all think?

Read the linked rust-tuf pull request and the review discussion around comment r512292123 first. Trace the two described update attempts and determine whether failed non-volatile storage writes should fail the workflow or only report persistence failure; done requires an agreed behavior reflected in the specification.