Graceful degration when dealing with remote storages

For historical reasons, most of our functions for working with block fractions assumed they were working with a local file system.

For example, if you look at the code of the following functions and methods, you can see that any error triggers panic() or logger.Fatal, which leads either to catching the panic and returning an HTTP 5XX to the client, or to process termination:

• Loader.Load;
• Loader.skipBlock;

Working with S3 requires a different approach to error handling, since this component is less reliable than the local file system and is subject to more frequent errors.

Let's divide errors into two types: soft-error and hard-error.

In my opinion, soft-error type errors do NOT indicate that the data is corrupted or simply missing.
They indicate that some subsystem was unable to process the request correctly and we will need to retry it in the future.

For example, soft-error errors can include:

• absence of network connection (for any reason);
• various network timeouts;

hard-error type errors indicate that the data has been corrupted and further work with such data will lead to violation of invariants in the program.
For such types of errors, the only correct thing to do is to forcibly stop the application, raise an alert, write a log, and require manual intervention for manual investigation.

For example, hard-error errors can include:

• absence of a fraction file in S3, although it should be there;
• reading some corrupted file;

The original definitions of "hard error" and "soft error" were for disks and go back to the early 1970s at least. A soft error was a recoverable error that could be handled by re-reading the disk sector. A hard error was a permanent disk error that could not be recovered.

Accordingly, it is proposed to think through this idea and implement it in the code.