Non-transient errors in reliable stream processing.

When performing high performance reliable stream processing using Orleans streams, persisting processed results for each event may lead to performance issues. In these cases, results tend to be kept in memory until a key event or time interval triggers persistence of a ‘checkpoint’. If an error occurs in the stream processing, the consumer may recover by loading the last checkpoint, rewinding the stream to the point where the checkpoint was taken, and then continuing normal processing from there.

This approach works well for recovering from transient errors, but it requires that the consumer know with certainty if an error is transient. If this recovery logic is attempted on a non-transient error (an event processing bug for instance), this could leave the stream processing logic in an infinite recovery loop.

It is reasonable to place the responsibility of detecting such infinite recovery loops on the application logic. The application will have a better understanding of the nature of the stream processing and how to handle unexpected errors. However, I’m raising this issue to explore capabilities Orleans streams could provide to help developers address this issue.

Start by reviewing Orleans streams' checkpoint, rewind, and recovery behavior described in the issue. Define whether Orleans should provide capabilities for detecting or preventing infinite recovery loops; the issue names no files, tests, entry points, or concrete acceptance condition.

Written by the indexing model from the issue text.