apache / apache/shenyu

[BUG] An error occurred using thread isolation mode in Hystrix Plugin

Open
#5,577 1 comment 0 reactions 0 assignees View on GitHub
type: bug
Dominant language
Java
Stars
8.8k
Forks
3.1k
Avg merge
7d 1h
Merged PRs (30d)
85

Description

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

### Use Shenyu V2.6.1 to enable Hystrix. When the isolation mode is semaphore, it is normal. When the isolation mode is thread, similar to this case: https://github.com/apache/shenyu/issues/2893,the error is as follows:
```
2024-05-31 15:27:40 [parallel-8] ERROR org.apache.shenyu.plugin.httpclient.AbstractHttpClientPlugin - Response took longer than timeout: PT3S
java.util.concurrent.TimeoutException: Response took longer than timeout: PT3S
at org.apache.shenyu.plugin.httpclient.AbstractHttpClientPlugin.lambda$execute$1(AbstractHttpClientPlugin.java:86)
at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:55)
at reactor.core.publisher.Mono.subscribe(Mono.java:4490)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:301)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:280)
at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:419)
at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2024-05-31 15:27:40 [hystrix-user2-4] ERROR org.apache.shenyu.plugin.hystrix.command.HystrixCommandOnThread - hystrix execute have error:
org.springframework.web.server.ResponseStatusException: 408 REQUEST_TIMEOUT "Request timeout, the maximum number of retry times has been exceeded"; nested exception is org.apache.shenyu.plugin.httpclient.exception.ShenyuTimeoutException: Request timeout, the maximum number of retry times has been exceeded
at org.apache.shenyu.plugin.httpclient.AbstractHttpClientPlugin.lambda$execute$5(AbstractHttpClientPlugin.java:100)
at reactor.core.publisher.Mono.lambda$onErrorMap$30(Mono.java:3797)
at reactor.core.publisher.Mono.lambda$onErrorResume$32(Mono.java:3887)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124)
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.whenError(FluxRetryWhen.java:225)
at reactor.core.publisher.FluxRetryWhen$RetryWhenOtherSubscriber.onError(FluxRetryWhen.java:274)
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:383)
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:251)
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
at reactor.core.publisher.EmitterProcessor.drain(EmitterProcessor.java:537)
at reactor.core.publisher.EmitterProcessor.tryEmitNext(EmitterProcessor.java:343)
at reactor.core.publisher.SinkManySerialized.tryEmitNext(SinkManySerialized.java:100)
at reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27)
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:190)
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258)
at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124)
at reactor.core.publisher.FluxTimeout$TimeoutOtherSubscriber.onError(FluxTimeout.java:341)
at reactor.core.publisher.Operators.error(Operators.java:198)
at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:56)
at reactor.core.publisher.Mono.subscribe(Mono.java:4490)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:301)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:280)
at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:419)
at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.shenyu.plugin.httpclient.exception.ShenyuTimeoutException: Request timeout, the maximum number of retry times has been exceeded
at org.apache.shenyu.plugin.httpclient.AbstractHttpClientPlugin.lambda$execute$4(AbstractHttpClientPlugin.java:97)
at reactor.util.retry.RetryBackoffSpec.lambda$null$4(RetryBackoffSpec.java:560)
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:376)
... 28 common frames omitted
```
**The configuration of semaphore is as follows:**
semaphore

**The thread configuration is as follows:**
thread

### Expected Behavior

Thread isolation mode should be normal

### Steps To Reproduce

1. OS:CentOS Linux 7 (Core),JDK:openjdk version "1.8.0_412"
2. The thread configuration is as follows
thread
3. Using Postman to initiate request calls
4. An error occurred

### Environment

```markdown
ShenYu version(s):v2.6.1
```

### Debug logs

_No response_

### Anything else?

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with AbstractHttpClientPlugin at the timeout stack-trace locations and HystrixCommandOnThread, then reproduce the v2.6.1 setup with thread isolation using Postman. Compare it with semaphore isolation and trace why the thread mode times out; done means thread isolation handles the request normally without the reported timeout errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.