Gluten support on external shuffle service
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
Now,
Gluten reported register issue, when set spark.shuffle.service.enabled=true;
```
25/03/26 14:44:37 ERROR YarnClusterScheduler: Lost executor 590 on 10.5.132.235: Unable to create executor due to Unable to register with external shuffle server due to : java.lang.UnsupportedOperationException: Unsupported shuffle manager of executor: ExecutorShuffleInfo[localDirs=[/data1/emr/yarn/local/usercache/jameswang/appcache/application_1655737030493_117316282/blockmgr-c0aa9150-7dd3-4c17-96cf-296377b216d6, /data/emr/yarn/local/usercache/jameswang/appcache/application_1655737030493_117316282/blockmgr-715c178d-c923-4028-9503-0cef887f9bd9],subDirsPerLocalDir=64,shuffleManager=org.apache.spark.shuffle.sort.ColumnarShuffleManager]
at org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.registerExecutor(ExternalShuffleBlockResolver.java:155)
```
Contributor guide
Assessment
This issue has not been assessed yet.