[Feature] Support short-circuit read in local-file
- Dominant language
- Java
- Stars
- 454
- Forks
- 172
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 5
Description
### Motivation
When uniffle shuffle servers are co-located with Yarn nodemanagers, we could use the short-circuit read to improve performance and reduce the overhead.
### How to do
There are two options to solve this
1. Directly read shuffle-data files by client side.
2. Use the domain socket.
#### Option 1 - Directly read
This is the fastest way, but the local-files' read permission should be open for client side. This maybe have security problem.
#### Option2 - Domain socket
This way could be avoid security problem, but will be slower than above. And its implementation will be complex.
### Conclusion
In my opinion, above two ways could be all supported in uniffle, which could be as different policies for users to choose.
Contributor guide
Research direction
The issue does not name any files, tests, or entry points. Begin by locating the local-file shuffle read path and its client/server boundary, then compare direct file access with a domain-socket design. Done should include a decided policy model, an implemented short-circuit path, and tests for the supported policy choices and permission behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100