grpc: requests erroring with upstream_reset_after_response_started{remote_reset} after timeout in envoy v1.31.0
- Dominant language
- C++
- Stars
- 28.9k
- Forks
- 5.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 437
Description
*Title*: *GRPC requests erroring with upstream_reset_after_response_started{remote_reset} after timeout in envoy v1.31.0*
*Description*:
Apologies if the repro steps here are a bit bare, we are still trying to find a reasonable repro, but I wanted to open an issue in case there was an obvious related discussion about this already that I missed. My ask is this: is there something that went out in 1.31.0 that may have caused issues with how envoy handles grpc?
We recently upgraded from envoy 1.30.3 to 1.31.0. After this release we saw this behavior in our mesh:
```
{
"attributes": {
"envoy_status_code": "200",
"bytes_received": 99,
"timing": {
"duration": 30001,
"request_duration": 0,
"response_duration": 7
},
"bytes_sent": 0,
"size": 2612,
"response": {
"response_code_details": "upstream_reset_after_response_started{remote_reset}",
"grpc-status": "-/-", # this format is "grpc-status": "%RESP(GRPC-STATUS)%/%TRAILER(GRPC-STATUS)%",
"response_code": 200,
"x-envoy-upstream-service-time": "6",
"flags": "UR"
},
}
```
grpc responses were erroring with upstream_reset_after_response_started{remote_reset} and timing out at the max timeout of 30000 ms
Normally these requests take <10ms and are not very large in bandwidth.
Reverting version to 1.30.3 seems to have "fixed" the issue. Our current running theory is that
* envoy saw headers from the upstream
* envoy was waiting for trailers, (thus the status is -/-) but given that the status is empty it looks like the trailer did not include the status code.
>What issue is being seen? Describe what should be happening instead of
the bug, for example: Envoy should not crash, the expected value isn't
returned, etc.
here's the log line of a "normal" request (relevant parts)
```
{
"attributes": {
"envoy_status_code": "200",
"bytes_received": 100,
"timing": {
"duration": 11,
"response_tx_duration": 0,
"request_duration": 0,
"response_duration": 11
},
"bytes_sent": 80,
"size": 2565,
"response": {
"response_code_details": "via_upstream",
"grpc-status": "-/0", # this format is "grpc-status": "%RESP(GRPC-STATUS)%/%TRAILER(GRPC-STATUS)%",
"response_code": 200,
"x-envoy-upstream-service-time": "10",
"flags": "-"
},
}
}
}
```
the client is grpc-java-netty/1.62.2
Contributor guide
Assessment
This issue has not been assessed yet.