envoyproxy / envoyproxy/envoy

grpc: requests erroring with upstream_reset_after_response_started{remote_reset} after timeout in envoy v1.31.0

Open
#36,188 13 comments 0 reactions 1 assignee Claimed by @shulin-sq View on GitHub
area/grpc bug no stalebot oghttp2
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 20h
Merged PRs (30d)
437

Description

*Title*: *GRPC requests erroring with upstream_reset_after_response_started{remote_reset} after timeout in envoy v1.31.0*

*Description*:
Apologies if the repro steps here are a bit bare, we are still trying to find a reasonable repro, but I wanted to open an issue in case there was an obvious related discussion about this already that I missed. My ask is this: is there something that went out in 1.31.0 that may have caused issues with how envoy handles grpc?

We recently upgraded from envoy 1.30.3 to 1.31.0. After this release we saw this behavior in our mesh:

```
{
"attributes": {
"envoy_status_code": "200",
"bytes_received": 99,
"timing": {
"duration": 30001,
"request_duration": 0,
"response_duration": 7
},
"bytes_sent": 0,
"size": 2612,
"response": {
"response_code_details": "upstream_reset_after_response_started{remote_reset}",
"grpc-status": "-/-", # this format is "grpc-status": "%RESP(GRPC-STATUS)%/%TRAILER(GRPC-STATUS)%",
"response_code": 200,
"x-envoy-upstream-service-time": "6",
"flags": "UR"
},
}
```

grpc responses were erroring with upstream_reset_after_response_started{remote_reset} and timing out at the max timeout of 30000 ms

Normally these requests take <10ms and are not very large in bandwidth.

Reverting version to 1.30.3 seems to have "fixed" the issue. Our current running theory is that
* envoy saw headers from the upstream
* envoy was waiting for trailers, (thus the status is -/-) but given that the status is empty it looks like the trailer did not include the status code.

>What issue is being seen? Describe what should be happening instead of
the bug, for example: Envoy should not crash, the expected value isn't
returned, etc.

here's the log line of a "normal" request (relevant parts)

```
{
"attributes": {
"envoy_status_code": "200",
"bytes_received": 100,
"timing": {
"duration": 11,
"response_tx_duration": 0,
"request_duration": 0,
"response_duration": 11
},
"bytes_sent": 80,
"size": 2565,
"response": {
"response_code_details": "via_upstream",
"grpc-status": "-/0", # this format is "grpc-status": "%RESP(GRPC-STATUS)%/%TRAILER(GRPC-STATUS)%",
"response_code": 200,
"x-envoy-upstream-service-time": "10",
"flags": "-"
},
}
}
}
```

the client is grpc-java-netty/1.62.2

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.