[BUG] An error occurred using thread isolation mode in Hystrix Plugin
- Dominant language
- Java
- Stars
- 8.8k
- Forks
- 3.1k
- Avg merge
- 7d 1h
- Merged PRs (30d)
- 85
Description
### Is there an existing issue for this?
- [X] I have searched the existing issues
### Current Behavior
### Use Shenyu V2.6.1 to enable Hystrix. When the isolation mode is semaphore, it is normal. When the isolation mode is thread, similar to this case: https://github.com/apache/shenyu/issues/2893,the error is as follows:
```
2024-05-31 15:27:40 [parallel-8] ERROR org.apache.shenyu.plugin.httpclient.AbstractHttpClientPlugin - Response took longer than timeout: PT3S
java.util.concurrent.TimeoutException: Response took longer than timeout: PT3S
at org.apache.shenyu.plugin.httpclient.AbstractHttpClientPlugin.lambda$execute$1(AbstractHttpClientPlugin.java:86)
at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:55)
at reactor.core.publisher.Mono.subscribe(Mono.java:4490)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:301)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:280)
at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:419)
at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2024-05-31 15:27:40 [hystrix-user2-4] ERROR org.apache.shenyu.plugin.hystrix.command.HystrixCommandOnThread - hystrix execute have error:
org.springframework.web.server.ResponseStatusException: 408 REQUEST_TIMEOUT "Request timeout, the maximum number of retry times has been exceeded"; nested exception is org.apache.shenyu.plugin.httpclient.exception.ShenyuTimeoutException: Request timeout, the maximum number of retry times has been exceeded
at org.apache.shenyu.plugin.httpclient.AbstractHttpClientPlugin.lambda$execute$5(AbstractHttpClientPlugin.java:100)
at reactor.core.publisher.Mono.lambda$onErrorMap$30(Mono.java:3797)
at reactor.core.publisher.Mono.lambda$onErrorResume$32(Mono.java:3887)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124)
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.whenError(FluxRetryWhen.java:225)
at reactor.core.publisher.FluxRetryWhen$RetryWhenOtherSubscriber.onError(FluxRetryWhen.java:274)
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:383)
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:251)
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
at reactor.core.publisher.EmitterProcessor.drain(EmitterProcessor.java:537)
at reactor.core.publisher.EmitterProcessor.tryEmitNext(EmitterProcessor.java:343)
at reactor.core.publisher.SinkManySerialized.tryEmitNext(SinkManySerialized.java:100)
at reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27)
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:190)
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258)
at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124)
at reactor.core.publisher.FluxTimeout$TimeoutOtherSubscriber.onError(FluxTimeout.java:341)
at reactor.core.publisher.Operators.error(Operators.java:198)
at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:56)
at reactor.core.publisher.Mono.subscribe(Mono.java:4490)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:301)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:280)
at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:419)
at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.shenyu.plugin.httpclient.exception.ShenyuTimeoutException: Request timeout, the maximum number of retry times has been exceeded
at org.apache.shenyu.plugin.httpclient.AbstractHttpClientPlugin.lambda$execute$4(AbstractHttpClientPlugin.java:97)
at reactor.util.retry.RetryBackoffSpec.lambda$null$4(RetryBackoffSpec.java:560)
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:376)
... 28 common frames omitted
```
**The configuration of semaphore is as follows:**
**The thread configuration is as follows:**
### Expected Behavior
Thread isolation mode should be normal
### Steps To Reproduce
1. OS:CentOS Linux 7 (Core),JDK:openjdk version "1.8.0_412"
2. The thread configuration is as follows
3. Using Postman to initiate request calls
4. An error occurred
### Environment
```markdown
ShenYu version(s):v2.6.1
```
### Debug logs
_No response_
### Anything else?
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with AbstractHttpClientPlugin at the timeout stack-trace locations and HystrixCommandOnThread, then reproduce the v2.6.1 setup with thread isolation using Postman. Compare it with semaphore isolation and trace why the thread mode times out; done means thread isolation handles the request normally without the reported timeout errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100