[Bug] Getting shuffle handle info failed due to the driver pressure
- Dominant language
- Java
- Stars
- 454
- Forks
- 172
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 5
Description
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
```
org.apache.uniffle.shaded.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: CallOptions deadline exceeded after 59.999985740s. Name resolution delay 0.000000000 seconds. [closed=[], open=[[remote_addr=nodexxxxxi.hadoop/10.xx.xx.xx:45921]]]
at org.apache.uniffle.shaded.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268)
at org.apache.uniffle.shaded.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249)
at org.apache.uniffle.shaded.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167)
at org.apache.uniffle.proto.ShuffleManagerGrpc$ShuffleManagerBlockingStub.getPartitionToShufflerServerWithBlockRetry(ShuffleManagerGrpc.java:644)
at org.apache.uniffle.client.impl.grpc.ShuffleManagerGrpcClient.getPartitionToShufflerServerWithBlockRetry(ShuffleManagerGrpcClient.java:109)
at org.apache.uniffle.shuffle.manager.RssShuffleManagerBase.getRemoteShuffleHandleInfoWithBlockRetry(RssShuffleManagerBase.java:955)
at org.apache.uniffle.shuffle.manager.RssShuffleManagerBase.getShuffleHandleInfo(RssShuffleManagerBase.java:912)
at org.apache.spark.shuffle.writer.RssShuffleWriter.(RssShuffleWriter.java:307)
at org.apache.spark.shuffle.RssShuffleManager.getWriter(RssShuffleManager.java:260)
at org.apache.spark.shuffle.QiyiRssShuffleManager.getWriter(QiyiRssShuffleManager.java:249)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:57)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
### Affects Version(s)
master
### Uniffle Server Log Output
```logtalk
```
### Uniffle Engine Log Output
```logtalk
```
### Uniffle Server Configurations
```yaml
```
### Uniffle Engine Configurations
```yaml
```
### Additional context
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
Contributor guide
Research direction
The failure path is shown through RssShuffleManagerBase.java, ShuffleManagerGrpcClient.java, and Spark's RssShuffleWriter.java; start by tracing getShuffleHandleInfoWithBlockRetry and the gRPC call that times out. Reproduce the failure under driver pressure and determine the expected behavior; done means the timeout is addressed with regression coverage. The issue provides no server or engine logs, configurations, or named tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, java, spark
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100