apache / apache/uniffle

[Bug] shuffle server have blocked threads

Open
#1,026 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

@jerqi @zuston

hi, when huge table join huge table, shuffle server have blocked threads , Is it right?

server conf :
rss.rpc.server.port 20000
rss.jetty.http.port 20001
rss.storage.basePath /app/rss-0.7.1/data
rss.storage.type MEMORY_LOCALFILE_HDFS
rss.coordinator.quorum 172.100.3.70:19999,172.100.3.71:19999,172.100.3.72:19999
rss.server.disk.capacity 50g

rss.server.flush.thread.alive 30
rss.server.flush.threadPool.size 10
rss.server.buffer.capacity 40g
rss.server.read.buffer.capacity 20g
rss.server.heartbeat.interval 10000
rss.rpc.message.max.size 1073741824
rss.server.preAllocation.expired 120000
rss.server.commit.timeout 600000
rss.server.app.expired.withoutHeartbeat 120000
rss.server.flush.cold.storage.threshold.size 512m

rss client conf :

spark.shuffle.manager=org.apache.spark.shuffle.RssShuffleManager
spark.rss.coordinator.quorum=172.100.3.70:19999,172.100.3.71:19999,172.100.3.72:19999

spark.rss.storage.type=MEMORY_LOCALFILE_HDFS
spark.rss.remote.storage.path=hdfs://ns1/rss/sml

![image](https://github.com/apache/incubator-uniffle/assets/3837738/4fdb9724-4e6d-497a-96dc-01ee45ec9f75)

![image](https://github.com/apache/incubator-uniffle/assets/3837738/97f9a59b-5909-4db6-aef0-9ac89805ab29)

![image](https://github.com/apache/incubator-uniffle/assets/3837738/3ed319e3-f1d9-417c-a570-b587ab70850e)

the executor have no daemon thread holding and hava no error log

![image](https://github.com/apache/incubator-uniffle/assets/3837738/1baad065-8769-4092-82b6-5dbb24478167)

![image](https://github.com/apache/incubator-uniffle/assets/3837738/76b81e9d-1729-40f0-9bec-2c0da7052a79)

### Affects Version(s)

0.7.1

### Uniffle Server Log Output

_No response_

### Uniffle Engine Log Output

_No response_

### Uniffle Server Configurations

_No response_

### Uniffle Engine Configurations

_No response_

### Additional context

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by examining the shuffle-server thread dump and server logs for the huge-table join, then compare the supplied server settings with the Spark RssShuffleManager configuration on Uniffle 0.7.1. Reproduce the blocked-thread state and capture the missing logs or stack traces; done means the blocking cause is identified and a focused fix or confirmed configuration change is covered by an appropriate regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.