Timeout exception on StopPrimarySilo when using persistent streams

Hi,

when stopping a primary silo (I have only two silos in my test) the execution hangs in `Silo.Terminate` method in step 9 (closing stream providers). This happens only when I use persistent streams (any).
I was able to track the issue to `GrainBasedPubSubRuntime.UnregisterProducer`, which should call `PubSubRendezvousGrain.UnregisterProducer` but this call never happens.

The following exception is logged when the method is called:
```
Silo S127.0.0.1:29668:232613388 is rejecting message: Request S127.0.0.1:29668:232613388Catalog@S0000000e->S127.0.0.1:29669:232613395DirectoryService@S0000000a #208: global::Orleans.Runtime.IRemoteGrainDirectory:LookupAsync(). Reason = Recent (00:00:00.0010001 ago, at 2017-05-16 06:56:06.488 GMT) connection failure trying to reach target silo S127.0.0.1:29669:232613395. Going to drop Request msg 208 without sending. CONNECTION_RETRY_DELAY = 00:00:01.

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__113.MoveNext() in C:\Users\MO185073\Projects\orleans-1.4.1\src\OrleansRuntime\GrainDirectory\LocalGrainDirectory.cs:line 895
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Orleans.Runtime.Scheduler.SchedulerExtensions.<>c__DisplayClass0_0`1.<b__0>d.MoveNext() in C:\Users\MO185073\Projects\orleans-1.4.1\src\OrleansRuntime\Scheduler\ShedulerExtensions.cs:line 16
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Orleans.Runtime.Placement.RandomPlacementDirector.d__1.MoveNext() in C:\Users\MO185073\Projects\orleans-1.4.1\src\OrleansRuntime\Placement\RandomPlacementDirector.cs:line 14
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Orleans.Runtime.Placement.PlacementDirectorsManager.d__6.MoveNext() in C:\Users\MO185073\Projects\orleans-1.4.1\src\OrleansRuntime\Placement\PlacementDirectorsManager.cs:line 59
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Orleans.Runtime.Dispatcher.d__30.MoveNext() in C:\Users\MO185073\Projects\orleans-1.4.1\src\OrleansRuntime\Core\Dispatcher.cs:line 585
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at Orleans.Runtime.Dispatcher.d__27.MoveNext() in C:\Users\MO185073\Projects\orleans-1.4.1\src\OrleansRuntime\Core\Dispatcher.cs:line 538
```

From this point forward nothing happens and I have to keep waiting for 10 minutes for the timeout exception.

After spending ton of time debugging and looking through the issues here I found a workaround, which works. Simply stop the stream provider before stopping the silos:
```
_cluster?
.GrainFactory
.GetGrain(0)
.SendControlCommandToProvider(
"Orleans.Streams.RabbitMqStreamProvider",
Globals.StreamProviderName,
(int) PersistentStreamProviderCommand.StopAgents)
.Wait();

_cluster?.StopAllSilos();
```

I think the call to `PubSubRendezvousGrain` requires messaging to work, but it was disabled in some earlier step in `Silo.Terminate`.

Is this expected, or a bug?

Thank you,
Martin

Start at Silo.Terminate step 9 and GrainBasedPubSubRuntime.UnregisterProducer, then inspect the PubSubRendezvousGrain.UnregisterProducer call and the LocalGrainDirectory.LookupAsync failure shown in the stack trace. Done means stopping silos with persistent streams no longer hangs or waits for the timeout exception, without requiring the stream provider workaround.

Written by the indexing model from the issue text.