[Bug] Triple Java client HTTP/2 flow-control failure with PHP gRPC unary response larger than 64 KiB
- Dominant language
- Java
- Stars
- 41.6k
- Forks
- 26.4k
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 4
Description
### Pre-check
- [x] I am sure that all the content I provide is in English.
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/dubbo/issues?q=is%3Aissue) and found no similar issues.
### Apache Dubbo Component
Java SDK (apache/dubbo)
### Dubbo Version
Environment 1:
- Apache Dubbo Java: 3.3.1
- Netty: 4.1.107.Final and 4.1.113.Final
- JDK: 1.8.0_192
- OS: Linux
- Consumer: Java application using the Triple protocol
- Provider: PHP application using standard gRPC over HTTP/2
Environment 2:
- Apache Dubbo Java: 3.3.7-ygf-SNAPSHOT
- Based on the Dubbo 3.3 branch and including the Triple backpressure changes from https://github.com/apache/dubbo/pull/15957
- Netty: 4.2.15.Final
- JDK: 1.8.0_192
- OS: Linux
- Consumer: Java application using the Triple protocol
- Provider: PHP application using standard gRPC over HTTP/2
### Steps to reproduce this issue
## Topology
```text
Dubbo Java Triple consumer (10.16.8.5)
|
| HTTP/2 + gRPC
v
PHP gRPC provider (10.16.8.8:50051)
```
The invoked unary method is:
```text
UserDataBusiness/ExportUserData
```
The serialized response is approximately 92 KiB:
```text
gRPC payload: approximately 94,486 bytes
gRPC record including the 5-byte prefix: approximately 94,491 bytes
```
The issue is reproducible when the PHP gRPC server returns this response.
## Case 1: 64 KiB initial stream window
Configure or advertise `SETTINGS_INITIAL_WINDOW_SIZE = 65535`.
The PHP server sends exactly the available stream-window data and then waits for a stream-level `WINDOW_UPDATE`. The Java consumer does not complete the approximately 92 KiB gRPC message and the call eventually times out.
Observed sequence:
```text
Java -> PHP: SETTINGS_INITIAL_WINDOW_SIZE = 65535
PHP -> Java: response DATA until the stream window is exhausted
PHP: waits for stream-level WINDOW_UPDATE
Java: waits for the remaining bytes of the complete gRPC message
Java -> PHP: RST_STREAM CANCEL after the RPC timeout
```
This looks like a flow-control progress deadlock: the decoder needs more DATA to complete the message, while the sender needs WINDOW_UPDATE to send the remaining DATA.
## Case 2: advertised window increased to 8 MiB
When the restriction is removed, the Java client advertises:
```text
SETTINGS_INITIAL_WINDOW_SIZE = 8,388,608
SETTINGS_MAX_FRAME_SIZE = 8,388,608
```
The packet capture confirms these client SETTINGS values:
```text
SETTINGS_HEADER_TABLE_SIZE = 1,048,576
SETTINGS_ENABLE_PUSH = 0
SETTINGS_MAX_CONCURRENT_STREAMS = 2,147,483,647
SETTINGS_INITIAL_WINDOW_SIZE = 8,388,608
SETTINGS_MAX_FRAME_SIZE = 8,388,608
SETTINGS_MAX_HEADER_LIST_SIZE = 32,768
```
The PHP server then sends the complete approximately 92 KiB response immediately, split into multiple HTTP/2 DATA frames (mostly 16 KiB each). The full response reaches the Java host in less than 100 ms.
However, the Java client fails while Netty processes the inbound DATA frames:
```text
io.netty.handler.codec.http2.Http2Exception$StreamException:
Flow control window exceeded for stream: 5
at io.netty.handler.codec.http2.Http2Exception.streamError(...)
at io.netty.handler.codec.http2.DefaultHttp2LocalFlowController$DefaultState.receiveFlowControlledFrame(...)
at io.netty.handler.codec.http2.DefaultHttp2LocalFlowController.receiveFlowControlledFrame(...)
at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(...)
```
The application finally receives:
```text
org.apache.dubbo.rpc.StatusRpcException:
CANCELLED : Canceled by remote peer, errorCode=2
```
According to the packet capture, the Java side sends the reset after the local Netty flow-control exception; the PHP provider does not cancel the RPC first.
The same exception is reproducible with official Dubbo 3.3.1:
```text
dubbo version: 3.3.1
netty-codec-http2: 4.1.113.Final
Flow control window exceeded for stream: 3
```
It was also observed with Netty 4.1.107.Final, so this does not appear to be exclusively introduced by PR #15957.
### What you expected to happen
The Java Triple consumer should successfully receive and decode a unary gRPC response of approximately 92 KiB.
The response is smaller than Dubbo's defaults:
```text
DEFAULT_INITIAL_WINDOW_SIZE = 8 MiB
DEFAULT_MAX_MESSAGE_SIZE = 50 MiB
```
The receive-window value advertised to the peer should be consistent with the window enforced by the Java client's local HTTP/2 flow controller.
If the client advertises `SETTINGS_INITIAL_WINDOW_SIZE = 8 MiB`, receiving approximately 92 KiB of DATA on one stream should not result in `Flow control window exceeded for stream`.
With a smaller window, the client should return flow-control credit as DATA is consumed:
```text
PHP sends part of the response
Java consumes the received DATA
Java sends stream-level WINDOW_UPDATE
PHP sends the remaining response
Java completes the unary RPC successfully
```
### Anything else
## Initial analysis
We currently see two related failure modes.
### 1. Progress problem with a 64 KiB window
If consumed bytes are returned only after a complete gRPC message has been assembled, a message larger than the stream window cannot make progress:
```text
Decoder needs more DATA to complete the message
Sender needs WINDOW_UPDATE to send more DATA
WINDOW_UPDATE depends on the decoder consuming the message
```
The backpressure work in https://github.com/apache/dubbo/pull/15957 appears related to this part.
### 2. Advertised and locally enforced windows may be inconsistent
When Java advertises an 8 MiB stream window, the PHP server legally sends the complete 92 KiB response. Netty nevertheless reports that the stream window is exceeded.
Possible explanations include:
- `Http2Settings.initialWindowSize` is sent to the peer but not consistently applied to the local `Http2LocalFlowController`.
- A newly created stream is initialized with 65,535 bytes instead of the configured value.
- Flow-controlled bytes arrive faster than the local consumption path updates the window.
- Stream-channel automatic/manual flow-control accounting is not synchronized.
- Configuration is applied to the outbound SETTINGS frame but not to every internal receive-window state.
These are hypotheses based on packet captures and stack traces, not a confirmed root cause.
## Why PHP gRPC exposes it consistently
The PHP gRPC implementation trusts the Java client's SETTINGS. If Java advertises 8 MiB, PHP may immediately send the complete response.
Java-to-Java Triple calls may use different write scheduling or backpressure behavior and may interleave transmission and consumption, hiding the same receive-window problem.
## Clarification about our usage pattern
We understand that our current usage pattern may not be ideal:
- A standard PHP gRPC provider is called by a Dubbo Java Triple consumer.
- The method returns a relatively large unary response.
- We experimented with HTTP/2 window settings as a workaround.
- Server streaming may be more appropriate if the response continues to grow.
However, the current response is only approximately 92 KiB and remains below Dubbo's default message/window limits.
We would appreciate guidance on:
1. Is a standard PHP gRPC provider officially supported as a Dubbo Java Triple interoperability scenario?
2. Should unary responses larger than 64 KiB work without application-level streaming?
3. Must users configure both stream and connection windows manually?
4. Should `TripleConfig.initialWindowSize` automatically initialize both the outbound SETTINGS value and the local `Http2LocalFlowController`?
5. Is PR #15957 expected to resolve this case completely, or only provide application-level backpressure APIs?
6. Would explicitly initializing the local controller be appropriate?
For example:
```java
int initialWindowSize = tripleConfig.getInitialWindowSizeOrDefault();
TripleHttp2LocalFlowController flowController =
new TripleHttp2LocalFlowController(connection, windowUpdateRatio);
flowController.initialWindowSize(initialWindowSize);
connection.local().flowController(flowController);
```
We can provide packet captures and prepare a minimal Java/PHP reproduction if needed.
### Do you have a (mini) reproduction demo?
- [ ] Yes, I have a minimal reproduction demo to help resolve this issue more effectively!
### Are you willing to submit a pull request to fix on your own?
- [ ] Yes I am willing to submit a pull request on my own!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start with TripleConfig.initialWindowSize, TripleHttp2LocalFlowController, and Netty's DefaultHttp2LocalFlowController path described in the stack trace; reproduce the 92 KiB PHP gRPC unary response with both 64 KiB and 8 MiB settings. Done means determining whether advertised and enforced windows diverge or consumption stalls, then showing the unary RPC completes without a flow-control exception or timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, java, php
- Domain
- backend-api-design, distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100