Improve retry algorithm for Tasks
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
Currently we just use exponential backoff. In practice when there are many concurrent writes, the conflict rate is still high even with jitter. There are some other algorithms like AIMD that was proven to be effective, and might be worth considering to add to Tasks to retry commits.
Reference: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-emrfs-retry.html
### Query engine
None
Contributor guide
Research direction
The issue does not name implementation files or tests. Start by locating the Tasks commit-retry path and documenting how its current exponential backoff and jitter work; compare possible alternatives with the linked AWS reference. Done requires an agreed retry algorithm and tests showing its behavior under concurrent writes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100