Client: graceful way to pause/continue and deal with runtime errors?

Hi there! Thanks for the great work on the OPTIMADE client, it's really pleasant to use, and looks very well designed. I appreciate it especially having written the (quite poorer) interface in pymatgen, which one day perhaps we can deprecate :)

While using the interface, I frequently encounter timeout errors with some databases. These might look like:

returned: [RuntimeError: 502 -

or

returned: ['RuntimeError: 500 -

This raises a few questions, and I'm not sure the best resolution:

1. If using --output-file, the output file will be empty when an error is encountered, even if many structures were successfully retrieved. I understand using a callback is probably the best option here.
2. I can't see a good way to automatically skip the first N pages, if they have already been successfully retrieved.
3. I'm not sure if these two return codes will trigger the automatic retry; it looks like this will only happen for a TimeoutError, and not a RuntimeError?
4. The automatic retry does not seem to have a backoff period. Some databases request waiting for 30s for example. Perhaps we could take inspiration from the retry library or similar, and have an automatically increasing sleep time after each retry?

Apologies if there is an existing issue for this. I did have a look but couldn't find one. If I'm mis-using the library and this is already supported somehow, I'd be glad to know! Thanks again!

Start by tracing the client paths for --output-file, callbacks, page retrieval, and automatic retry handling for TimeoutError and RuntimeError. The issue needs a settled design before implementation; done should cover resumable output, page skipping, applicable errors, and increasing retry delays.