grpc / grpc/grpc

Unexpected timeouts in gevent environment

Open
#22,526 2 comments 0 reactions 0 assignees View on GitHub
disposition/help wanted kind/bug lang/Python priority/P2
Dominant language
C++
Stars
45.3k
Forks
11.4k
Avg merge
2d 12h
Merged PRs (30d)
47

Description

### What version of gRPC and what language are you using?
```
Python
grpcio 1.26.0
gevent 1.4.0
```

### What operating system (Linux, Windows,...) and version?

```
Linux
Debian 10
Ubuntu 19.04
```

### What runtime / compiler are you using (e.g. python version or version of gcc)

```
Python 3.7.4
```

### What did you do?
I defined a simple gRPC interface with a Go service implementing the server and a Python service invoking the gRPC calls. I defined a timeout on the call and set it to 500ms:

```
stub.Query(request, timeout=0.5)
```

### What did you expect to see?

I don't expect the request to ever timeout since the round-trip latency has a p99 of 40ms.

### What did you see instead?

A certain percentage of requests are timing out raising a `DEADLINE_EXCEEDED` exception. The metrics don't go anywhere near the 500ms. Also when measuring the execution with `time.time()` the timeout happens after < 100ms.

```
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
E status = StatusCode.DEADLINE_EXCEEDED
E details = "Deadline Exceeded"
E debug_error_string = "{"created":"@1585697120.778573461","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":4}"
E >

../../.virtualenvs/streaming_api/lib/python3.7/site-packages/grpc/_channel.py:726: _InactiveRpcError
```

### Anything else we should know about your project / environment?

The Python service runs with gevent. I have applied the experimental gevent monkey patch:

```
import grpc.experimental.gevent as grpc_gevent
grpc_gevent.init_gevent()
```
Before applying it I experienced the timeouts as well. After applying it, it seems to happen more frequently now, even in a local test environment.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the timeout with Python 3.7.4, grpcio 1.26.0, gevent 1.4.0, and grpc.experimental.gevent.init_gevent(), using the stub.Query call described here. Inspect the deadline handling associated with src/core/ext/filters/deadline/deadline_filter.cc and the grpc/_channel.py traceback, then verify that a 500ms timeout is not reported before the request reaches that deadline.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, python
Domain
api, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.