apache / apache/gluten

[VL] LowCopyNettyJniByteInputStream does not work in uniffle netty mode

Open
#10,923 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

Shuffle reader does not use `LowCopyNettyJniByteInputStream` as expected in uniffle netty mode, but still uses `OnHeapJniByteInputStream`.

uniffle client conf:
```
spark.rss.client.type=GRPC_NETTY
```

### Gluten version

None

Contributor guide

Open the contributing guide

Research direction

Start by tracing the shuffle reader's input-stream selection when spark.rss.client.type=GRPC_NETTY, comparing where LowCopyNettyJniByteInputStream and OnHeapJniByteInputStream are chosen. Done means uniffle netty mode selects the low-copy stream as described and the behavior is covered by an appropriate regression check.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, scala, spark
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.