apache / apache/uniffle

[Feature] Support short-circuit read in local-file

Open
#172 18 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

### Motivation
When uniffle shuffle servers are co-located with Yarn nodemanagers, we could use the short-circuit read to improve performance and reduce the overhead.

### How to do
There are two options to solve this
1. Directly read shuffle-data files by client side.
2. Use the domain socket.

#### Option 1 - Directly read
This is the fastest way, but the local-files' read permission should be open for client side. This maybe have security problem.

#### Option2 - Domain socket
This way could be avoid security problem, but will be slower than above. And its implementation will be complex.

### Conclusion
In my opinion, above two ways could be all supported in uniffle, which could be as different policies for users to choose.

Contributor guide

Open the contributing guide

Research direction

The issue does not name any files, tests, or entry points. Begin by locating the local-file shuffle read path and its client/server boundary, then compare direct file access with a domain-socket design. Done should include a decided policy model, an implemented short-circuit path, and tests for the supported policy choices and permission behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.