[Bug] Concurrency limit filters leak active counts on downstream limit exceptions
- Dominant language
- Java
- Stars
- 41.6k
- Forks
- 26.4k
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 4
Description
### Pre-check
- [x] All content is in English.
- [x] I searched existing issues and pull requests for ActiveLimitFilter, ExecuteLimitFilter, active count, and limit-exception cleanup.
### Apache Dubbo Component
Java SDK (apache/dubbo), dubbo-rpc-api filters and the dubbo-cluster filter chain.
### Dubbo Version
3.3.7-SNAPSHOT, branch `3.3` at `dab47b7843`; Windows, OpenJDK 17.0.17, Maven 3.9.4.
### Steps to reproduce this issue
Both concurrency filters skip `RpcStatus.endCount()` whenever their error listener receives a `RpcException` with `LIMIT_EXCEEDED_EXCEPTION`. This correctly skips cleanup when the filter itself rejects admission, but also skips it when admission succeeded and a downstream invoker throws that exception (or its response future completes exceptionally with it).
A minimal synchronous reproduction in a test in `dubbo-cluster` (JUnit 5 / Mockito; ordinary imports omitted):
```java
URL url = URL.valueOf("test://localhost:12345/limit-repro?actives=1&timeout=1");
Invoker target = mock(Invoker.class);
when(target.getUrl()).thenReturn(url);
when(target.getInterface()).thenReturn(Object.class);
when(target.invoke(any())).thenThrow(
new RpcException(RpcException.LIMIT_EXCEEDED_EXCEPTION, "downstream rejected"));
Filter filter = new ActiveLimitFilter();
Invoker chain = new FilterChainBuilder.CallbackRegistrationInvoker<>(
new FilterChainBuilder.CopyOfFilterChainNode<>(target, target, filter),
Collections.singletonList(filter));
RpcInvocation invocation = new RpcInvocation();
invocation.setMethodName("invoke");
assertThrows(RpcException.class, () -> chain.invoke(invocation));
assertEquals(0, RpcStatus.getStatus(url, "invoke").getActive()); // actual: 1
```
Replace `actives=1` with `executes=1` and use `ExecuteLimitFilter` to reproduce the provider-side case.
For the asynchronous case, return `new AsyncRpcResult(pending, invocation)` from the target and complete `pending` exceptionally with the same limit exception after calling the chain. The active count also remains 1.
### What you expected to happen
A call that acquired a concurrency slot must release it on downstream failure and record the failure. A local admission rejection must not release another in-flight call's slot.
With a limit of 1, the leaked count causes later active-limit calls to time out or execute-limit calls to be rejected even though the failed call is no longer running.
### Verification
I added eight parameterized regression cases through the production `CopyOfFilterChainNode` / `CallbackRegistrationInvoker` lifecycle. Before the fix, all four downstream-failure cases fail with `expected: <0> but was: <1>`; the four local-rejection / invocation-reuse controls pass. There are 0 test errors.
The fix will track whether this filter acquired a slot for the current invocation attempt, resetting that state on each entry so sequential Invocation reuse does not carry stale admission state.
This concerns exceptions delivered through `onError`, not ordinary business exceptions carried inside an otherwise successfully completed `AppResponse`. No registry, network service, or load generator is needed for the reproduction.
AI assistance was used for source inspection, implementation, and regression tests. The failures above were reproduced locally, and I will submit the fix with the complete tests.
Contributor guide
Research direction
Start with ActiveLimitFilter and ExecuteLimitFilter, then trace the production CopyOfFilterChainNode and CallbackRegistrationInvoker lifecycle described in the issue. Run the dubbo-cluster JUnit 5 regression cases for synchronous and asynchronous downstream limit exceptions. Done means acquired slots are released and failures recorded, while local admission rejections and reused invocations remain correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100