Deadlock during close in Python client

### What version of gRPC and what language are you using?
1.17.1 Python

### What operating system (Linux, Windows,...) and version?
Linux 3.19.3.28478

### What runtime / compiler are you using (e.g. python version or version of gcc)
Python 3.5.3

### What did you do?
If possible, provide a recipe for reproducing the error. Try being specific and include code snippets if helpful.

I haven't been able to find a good way to consistently reproduce the issue. Originally, I thought that unhandled exceptions causing the channel spin thread to close were causing the issue as I found in #19910 , but that is not what is going on because the channel spin thread is still going. The situation we have is a controller device is running the client and a peripheral device is running the grpc server. A channel is opened on the client side and a function subscribes to the connectivity changes. When connectivity issues (wifi for instance) occur, the channel state goes to IDLE from READY and we close the channel so that we can try service discovery again. However, the channel is deadlocking in the close method which is being invoked from the main Python thread. Unfortunately, I don't have any knowledge of any exceptions that might be happening other than on the main Python thread, but I do have a stack trace.

We have a long-running unary_stream being iterated over in a separate thread. When it fails, we retry a few times before giving up. There are also two unary_unary streams. One is used for sending requests and the other for sending events. All requests/events on these two channels are sent using the future method on each stream with a timeout of either 1 second or 5 seconds.

We have the channel keepalive timeout set to 90 seconds and the keepalive time set to 20 seconds. keepalive is permitted even without calls. min time between pings is 8 seconds. max_pings_without_data is set to 0. We are seeing a keepalive timeout happen when the channel connectivity goes to IDLE and the main thread tries to close the channel. Several requests fail around this time with the keepalive timeout exception.

The stack trace is very interesting. The close method is definitely waiting on the channel condition variable so it is either waiting for segregated calls, integrated calls, or connectivity polling to finish, but due to Cython I don't know which.

The unary_stream thread is waiting on the channel condition variable as well. I'm guessing that the connection disruption was somewhat temporary in this case and that this unary_stream was able to be recreated.

There is also a thread polling connectivity which is interesting because we call unsubscribe before calling close on the channel. The polling connectivity thread is in _poll_connectivity in _channel.py calling channel.watch_connectivity_state. Unfortunately the Cython code is preventing me from getting more insight here.

The last relevant thread is the channel spin thread that handles responses for unary_unary streams. It is still running and calling state.channel.next_call_event. Again, Cython keeps me from knowing more. The fact that the channel spin thread is still going means that there is still at least one open request.

One other thing to note is that we have disabled grpc forking support because it was causing us deadlock problems due to other parts of the Python app doing some forking.

To me this is clearly a problem that is either in the channel_spin code or the watch_connectivity_state code. If the problem is in watch_connectivity_state, I suspect that it is with grpc_channel_watch_connectivity_state or _latent_event not returning. If the problem is in _next_call_event, then it is probably held up in _latent_event or the on_success callback which calls _process_integrated_call_tag.

It has occurred to me that both of these functions use _latent_event with a deadline of None.

### What did you expect to see?
The channel close would not deadlock.

### What did you see instead?
The channel close deadlocked.

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

See [TROUBLESHOOTING.md](https://github.com/grpc/grpc/blob/master/TROUBLESHOOTING.md) for how to diagnose problems better.

### Anything else we should know about your project / environment?