Unexpected timeouts in gevent environment
- Dominant language
- C++
- Stars
- 45.3k
- Forks
- 11.4k
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 47
Description
### What version of gRPC and what language are you using?
```
Python
grpcio 1.26.0
gevent 1.4.0
```
### What operating system (Linux, Windows,...) and version?
```
Linux
Debian 10
Ubuntu 19.04
```
### What runtime / compiler are you using (e.g. python version or version of gcc)
```
Python 3.7.4
```
### What did you do?
I defined a simple gRPC interface with a Go service implementing the server and a Python service invoking the gRPC calls. I defined a timeout on the call and set it to 500ms:
```
stub.Query(request, timeout=0.5)
```
### What did you expect to see?
I don't expect the request to ever timeout since the round-trip latency has a p99 of 40ms.
### What did you see instead?
A certain percentage of requests are timing out raising a `DEADLINE_EXCEEDED` exception. The metrics don't go anywhere near the 500ms. Also when measuring the execution with `time.time()` the timeout happens after < 100ms.
```
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
E status = StatusCode.DEADLINE_EXCEEDED
E details = "Deadline Exceeded"
E debug_error_string = "{"created":"@1585697120.778573461","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":4}"
E >
../../.virtualenvs/streaming_api/lib/python3.7/site-packages/grpc/_channel.py:726: _InactiveRpcError
```
### Anything else we should know about your project / environment?
The Python service runs with gevent. I have applied the experimental gevent monkey patch:
```
import grpc.experimental.gevent as grpc_gevent
grpc_gevent.init_gevent()
```
Before applying it I experienced the timeouts as well. After applying it, it seems to happen more frequently now, even in a local test environment.
Contributor guide
Research direction
Start by reproducing the timeout with Python 3.7.4, grpcio 1.26.0, gevent 1.4.0, and grpc.experimental.gevent.init_gevent(), using the stub.Query call described here. Inspect the deadline handling associated with src/core/ext/filters/deadline/deadline_filter.cc and the grpc/_channel.py traceback, then verify that a 500ms timeout is not reported before the request reaches that deadline.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, python
- Domain
- api, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100