apache / apache/dubbo

[Bug] Concurrency limit filters leak active counts on downstream limit exceptions

Open
#16,455 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
41.6k
Forks
26.4k
Avg merge
15h 13m
Merged PRs (30d)
4

Description

### Pre-check

- [x] All content is in English.
- [x] I searched existing issues and pull requests for ActiveLimitFilter, ExecuteLimitFilter, active count, and limit-exception cleanup.

### Apache Dubbo Component

Java SDK (apache/dubbo), dubbo-rpc-api filters and the dubbo-cluster filter chain.

### Dubbo Version

3.3.7-SNAPSHOT, branch `3.3` at `dab47b7843`; Windows, OpenJDK 17.0.17, Maven 3.9.4.

### Steps to reproduce this issue

Both concurrency filters skip `RpcStatus.endCount()` whenever their error listener receives a `RpcException` with `LIMIT_EXCEEDED_EXCEPTION`. This correctly skips cleanup when the filter itself rejects admission, but also skips it when admission succeeded and a downstream invoker throws that exception (or its response future completes exceptionally with it).

A minimal synchronous reproduction in a test in `dubbo-cluster` (JUnit 5 / Mockito; ordinary imports omitted):

```java
URL url = URL.valueOf("test://localhost:12345/limit-repro?actives=1&timeout=1");
Invoker target = mock(Invoker.class);
when(target.getUrl()).thenReturn(url);
when(target.getInterface()).thenReturn(Object.class);
when(target.invoke(any())).thenThrow(
new RpcException(RpcException.LIMIT_EXCEEDED_EXCEPTION, "downstream rejected"));

Filter filter = new ActiveLimitFilter();
Invoker chain = new FilterChainBuilder.CallbackRegistrationInvoker<>(
new FilterChainBuilder.CopyOfFilterChainNode<>(target, target, filter),
Collections.singletonList(filter));
RpcInvocation invocation = new RpcInvocation();
invocation.setMethodName("invoke");

assertThrows(RpcException.class, () -> chain.invoke(invocation));
assertEquals(0, RpcStatus.getStatus(url, "invoke").getActive()); // actual: 1
```

Replace `actives=1` with `executes=1` and use `ExecuteLimitFilter` to reproduce the provider-side case.

For the asynchronous case, return `new AsyncRpcResult(pending, invocation)` from the target and complete `pending` exceptionally with the same limit exception after calling the chain. The active count also remains 1.

### What you expected to happen

A call that acquired a concurrency slot must release it on downstream failure and record the failure. A local admission rejection must not release another in-flight call's slot.

With a limit of 1, the leaked count causes later active-limit calls to time out or execute-limit calls to be rejected even though the failed call is no longer running.

### Verification

I added eight parameterized regression cases through the production `CopyOfFilterChainNode` / `CallbackRegistrationInvoker` lifecycle. Before the fix, all four downstream-failure cases fail with `expected: <0> but was: <1>`; the four local-rejection / invocation-reuse controls pass. There are 0 test errors.

The fix will track whether this filter acquired a slot for the current invocation attempt, resetting that state on each entry so sequential Invocation reuse does not carry stale admission state.

This concerns exceptions delivered through `onError`, not ordinary business exceptions carried inside an otherwise successfully completed `AppResponse`. No registry, network service, or load generator is needed for the reproduction.

AI assistance was used for source inspection, implementation, and regression tests. The failures above were reproduced locally, and I will submit the fix with the complete tests.

Contributor guide

Open the contributing guide

Research direction

Start with ActiveLimitFilter and ExecuteLimitFilter, then trace the production CopyOfFilterChainNode and CallbackRegistrationInvoker lifecycle described in the issue. Run the dubbo-cluster JUnit 5 regression cases for synchronous and asynchronous downstream limit exceptions. Done means acquired slots are released and failures recorded, while local admission rejections and reused invocations remain correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.