[Bug] shuffle server have blocked threads
- Dominant language
- Java
- Stars
- 454
- Forks
- 172
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 5
Description
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
@jerqi @zuston
hi, when huge table join huge table, shuffle server have blocked threads , Is it right?
server conf :
rss.rpc.server.port 20000
rss.jetty.http.port 20001
rss.storage.basePath /app/rss-0.7.1/data
rss.storage.type MEMORY_LOCALFILE_HDFS
rss.coordinator.quorum 172.100.3.70:19999,172.100.3.71:19999,172.100.3.72:19999
rss.server.disk.capacity 50g
rss.server.flush.thread.alive 30
rss.server.flush.threadPool.size 10
rss.server.buffer.capacity 40g
rss.server.read.buffer.capacity 20g
rss.server.heartbeat.interval 10000
rss.rpc.message.max.size 1073741824
rss.server.preAllocation.expired 120000
rss.server.commit.timeout 600000
rss.server.app.expired.withoutHeartbeat 120000
rss.server.flush.cold.storage.threshold.size 512m
rss client conf :
spark.shuffle.manager=org.apache.spark.shuffle.RssShuffleManager
spark.rss.coordinator.quorum=172.100.3.70:19999,172.100.3.71:19999,172.100.3.72:19999
spark.rss.storage.type=MEMORY_LOCALFILE_HDFS
spark.rss.remote.storage.path=hdfs://ns1/rss/sml



the executor have no daemon thread holding and hava no error log


### Affects Version(s)
0.7.1
### Uniffle Server Log Output
_No response_
### Uniffle Engine Log Output
_No response_
### Uniffle Server Configurations
_No response_
### Uniffle Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
Contributor guide
Research direction
Start by examining the shuffle-server thread dump and server logs for the huge-table join, then compare the supplied server settings with the Spark RssShuffleManager configuration on Uniffle 0.7.1. Reproduce the blocked-thread state and capture the missing logs or stack traces; done means the blocking cause is identified and a focused fix or confirmed configuration change is covered by an appropriate regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100