apache / apache/gluten

Gluten support on external shuffle service

Open
#9,151 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Now,

Gluten reported register issue, when set spark.shuffle.service.enabled=true;

```
25/03/26 14:44:37 ERROR YarnClusterScheduler: Lost executor 590 on 10.5.132.235: Unable to create executor due to Unable to register with external shuffle server due to : java.lang.UnsupportedOperationException: Unsupported shuffle manager of executor: ExecutorShuffleInfo[localDirs=[/data1/emr/yarn/local/usercache/jameswang/appcache/application_1655737030493_117316282/blockmgr-c0aa9150-7dd3-4c17-96cf-296377b216d6, /data/emr/yarn/local/usercache/jameswang/appcache/application_1655737030493_117316282/blockmgr-715c178d-c923-4028-9503-0cef887f9bd9],subDirsPerLocalDir=64,shuffleManager=org.apache.spark.shuffle.sort.ColumnarShuffleManager]
at org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.registerExecutor(ExternalShuffleBlockResolver.java:155)

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.