Azure Storage Streaming, poison handling when message batch can not be deserialized

Hi,

Using the azure storage streaming provider with the default serializer causes an infinite retry loop of message batches serialized with the previous version.

The problem is most likely in https://github.com/dotnet/orleans/blob/bdaf800326b90c5e0950be6b0a8568add56f3b2c/src/Azure/Orleans.Streaming.AzureStorage/Providers/Streams/AzureQueue/AzureQueueAdapterReceiver.cs#L92

The batchcontainer can not be deserialized and should be considered as `poison`, but instead it keeps trying to read the same batch of events until Expiration Time is reached (which is 7 days by default in azure storage queues).

This situation can be reproduced using the following steps:

1. publish batch of message on a stream
2. shutdown silo (check if message is still in queue)
3. add property to the message
4. start silo with new message format

result is, `AzureQueueAdapterReceiver` throwing at https://github.com/dotnet/orleans/blob/bdaf800326b90c5e0950be6b0a8568add56f3b2c/src/Azure/Orleans.Streaming.AzureStorage/Providers/Streams/AzureQueue/AzureQueueAdapterReceiver.cs#L85

which in turn will throw at https://github.com/dotnet/orleans/blob/fd7375eebabd66e3826cd77710b1df13d9c0d3be/src/Orleans.Runtime/Streams/PersistentStream/PersistentStreamPullingAgent.cs#L435

which will trigger the retry from ` AsyncExecutorWithRetries`
In the next try, the previous batch is still invisible (due to InvisibilityTimeout, set when retrieving the faulty batch) so it will succeed.

Eventually our silo was only reading faulty batches and not processing any new messages.

We know that we shouldn't break contracts and that the default Orleans serializer is not version tolerant. We try to avoid making breaking changes, but missed one. It should not have broken down the entire silo I think

Start in src/Azure/Orleans.Streaming.AzureStorage/Providers/Streams/AzureQueue/AzureQueueAdapterReceiver.cs at the deserialization failure, then trace the retry through PersistentStreamPullingAgent.cs and AsyncExecutorWithRetries. Reproduce the version-mismatch scenario described in the issue and verify that an undecodable batch is treated as poison rather than retried until queue expiration.

Written by the indexing model from the issue text.