apache / apache/dubbo

[Bug] Triple Java client HTTP/2 flow-control failure with PHP gRPC unary response larger than 64 KiB

Open
#16,427 0 comments 0 reactions 0 assignees View on GitHub
component/need-triage type/need-triage
Dominant language
Java
Stars
41.6k
Forks
26.4k
Avg merge
15h 13m
Merged PRs (30d)
4

Description

### Pre-check

- [x] I am sure that all the content I provide is in English.

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/dubbo/issues?q=is%3Aissue) and found no similar issues.

### Apache Dubbo Component

Java SDK (apache/dubbo)

### Dubbo Version

Environment 1:
- Apache Dubbo Java: 3.3.1
- Netty: 4.1.107.Final and 4.1.113.Final
- JDK: 1.8.0_192
- OS: Linux
- Consumer: Java application using the Triple protocol
- Provider: PHP application using standard gRPC over HTTP/2

Environment 2:
- Apache Dubbo Java: 3.3.7-ygf-SNAPSHOT
- Based on the Dubbo 3.3 branch and including the Triple backpressure changes from https://github.com/apache/dubbo/pull/15957
- Netty: 4.2.15.Final
- JDK: 1.8.0_192
- OS: Linux
- Consumer: Java application using the Triple protocol
- Provider: PHP application using standard gRPC over HTTP/2

### Steps to reproduce this issue

## Topology

```text
Dubbo Java Triple consumer (10.16.8.5)
|
| HTTP/2 + gRPC
v
PHP gRPC provider (10.16.8.8:50051)
```

The invoked unary method is:

```text
UserDataBusiness/ExportUserData
```

The serialized response is approximately 92 KiB:

```text
gRPC payload: approximately 94,486 bytes
gRPC record including the 5-byte prefix: approximately 94,491 bytes
```

The issue is reproducible when the PHP gRPC server returns this response.

## Case 1: 64 KiB initial stream window

Configure or advertise `SETTINGS_INITIAL_WINDOW_SIZE = 65535`.

The PHP server sends exactly the available stream-window data and then waits for a stream-level `WINDOW_UPDATE`. The Java consumer does not complete the approximately 92 KiB gRPC message and the call eventually times out.

Observed sequence:

```text
Java -> PHP: SETTINGS_INITIAL_WINDOW_SIZE = 65535
PHP -> Java: response DATA until the stream window is exhausted
PHP: waits for stream-level WINDOW_UPDATE
Java: waits for the remaining bytes of the complete gRPC message
Java -> PHP: RST_STREAM CANCEL after the RPC timeout
```

This looks like a flow-control progress deadlock: the decoder needs more DATA to complete the message, while the sender needs WINDOW_UPDATE to send the remaining DATA.

## Case 2: advertised window increased to 8 MiB

When the restriction is removed, the Java client advertises:

```text
SETTINGS_INITIAL_WINDOW_SIZE = 8,388,608
SETTINGS_MAX_FRAME_SIZE = 8,388,608
```

The packet capture confirms these client SETTINGS values:

```text
SETTINGS_HEADER_TABLE_SIZE = 1,048,576
SETTINGS_ENABLE_PUSH = 0
SETTINGS_MAX_CONCURRENT_STREAMS = 2,147,483,647
SETTINGS_INITIAL_WINDOW_SIZE = 8,388,608
SETTINGS_MAX_FRAME_SIZE = 8,388,608
SETTINGS_MAX_HEADER_LIST_SIZE = 32,768
```

The PHP server then sends the complete approximately 92 KiB response immediately, split into multiple HTTP/2 DATA frames (mostly 16 KiB each). The full response reaches the Java host in less than 100 ms.

However, the Java client fails while Netty processes the inbound DATA frames:

```text
io.netty.handler.codec.http2.Http2Exception$StreamException:
Flow control window exceeded for stream: 5
at io.netty.handler.codec.http2.Http2Exception.streamError(...)
at io.netty.handler.codec.http2.DefaultHttp2LocalFlowController$DefaultState.receiveFlowControlledFrame(...)
at io.netty.handler.codec.http2.DefaultHttp2LocalFlowController.receiveFlowControlledFrame(...)
at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(...)
```

The application finally receives:

```text
org.apache.dubbo.rpc.StatusRpcException:
CANCELLED : Canceled by remote peer, errorCode=2
```

According to the packet capture, the Java side sends the reset after the local Netty flow-control exception; the PHP provider does not cancel the RPC first.

The same exception is reproducible with official Dubbo 3.3.1:

```text
dubbo version: 3.3.1
netty-codec-http2: 4.1.113.Final
Flow control window exceeded for stream: 3
```

It was also observed with Netty 4.1.107.Final, so this does not appear to be exclusively introduced by PR #15957.

### What you expected to happen

The Java Triple consumer should successfully receive and decode a unary gRPC response of approximately 92 KiB.

The response is smaller than Dubbo's defaults:

```text
DEFAULT_INITIAL_WINDOW_SIZE = 8 MiB
DEFAULT_MAX_MESSAGE_SIZE = 50 MiB
```

The receive-window value advertised to the peer should be consistent with the window enforced by the Java client's local HTTP/2 flow controller.

If the client advertises `SETTINGS_INITIAL_WINDOW_SIZE = 8 MiB`, receiving approximately 92 KiB of DATA on one stream should not result in `Flow control window exceeded for stream`.

With a smaller window, the client should return flow-control credit as DATA is consumed:

```text
PHP sends part of the response
Java consumes the received DATA
Java sends stream-level WINDOW_UPDATE
PHP sends the remaining response
Java completes the unary RPC successfully
```

### Anything else

## Initial analysis

We currently see two related failure modes.

### 1. Progress problem with a 64 KiB window

If consumed bytes are returned only after a complete gRPC message has been assembled, a message larger than the stream window cannot make progress:

```text
Decoder needs more DATA to complete the message
Sender needs WINDOW_UPDATE to send more DATA
WINDOW_UPDATE depends on the decoder consuming the message
```

The backpressure work in https://github.com/apache/dubbo/pull/15957 appears related to this part.

### 2. Advertised and locally enforced windows may be inconsistent

When Java advertises an 8 MiB stream window, the PHP server legally sends the complete 92 KiB response. Netty nevertheless reports that the stream window is exceeded.

Possible explanations include:

- `Http2Settings.initialWindowSize` is sent to the peer but not consistently applied to the local `Http2LocalFlowController`.
- A newly created stream is initialized with 65,535 bytes instead of the configured value.
- Flow-controlled bytes arrive faster than the local consumption path updates the window.
- Stream-channel automatic/manual flow-control accounting is not synchronized.
- Configuration is applied to the outbound SETTINGS frame but not to every internal receive-window state.

These are hypotheses based on packet captures and stack traces, not a confirmed root cause.

## Why PHP gRPC exposes it consistently

The PHP gRPC implementation trusts the Java client's SETTINGS. If Java advertises 8 MiB, PHP may immediately send the complete response.

Java-to-Java Triple calls may use different write scheduling or backpressure behavior and may interleave transmission and consumption, hiding the same receive-window problem.

## Clarification about our usage pattern

We understand that our current usage pattern may not be ideal:

- A standard PHP gRPC provider is called by a Dubbo Java Triple consumer.
- The method returns a relatively large unary response.
- We experimented with HTTP/2 window settings as a workaround.
- Server streaming may be more appropriate if the response continues to grow.

However, the current response is only approximately 92 KiB and remains below Dubbo's default message/window limits.

We would appreciate guidance on:

1. Is a standard PHP gRPC provider officially supported as a Dubbo Java Triple interoperability scenario?
2. Should unary responses larger than 64 KiB work without application-level streaming?
3. Must users configure both stream and connection windows manually?
4. Should `TripleConfig.initialWindowSize` automatically initialize both the outbound SETTINGS value and the local `Http2LocalFlowController`?
5. Is PR #15957 expected to resolve this case completely, or only provide application-level backpressure APIs?
6. Would explicitly initializing the local controller be appropriate?

For example:

```java
int initialWindowSize = tripleConfig.getInitialWindowSizeOrDefault();
TripleHttp2LocalFlowController flowController =
new TripleHttp2LocalFlowController(connection, windowUpdateRatio);
flowController.initialWindowSize(initialWindowSize);
connection.local().flowController(flowController);
```

We can provide packet captures and prepare a minimal Java/PHP reproduction if needed.

### Do you have a (mini) reproduction demo?

- [ ] Yes, I have a minimal reproduction demo to help resolve this issue more effectively!

### Are you willing to submit a pull request to fix on your own?

- [ ] Yes I am willing to submit a pull request on my own!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start with TripleConfig.initialWindowSize, TripleHttp2LocalFlowController, and Netty's DefaultHttp2LocalFlowController path described in the stack trace; reproduce the 92 KiB PHP gRPC unary response with both 64 KiB and 8 MiB settings. Done means determining whether advertised and enforced windows diverge or consumption stalls, then showing the unary RPC completes without a flow-control exception or timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, java, php
Domain
backend-api-design, distributed-systems, networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.