box / box/ClusterRunner

Network request backoff not coordinated across multiple threads

Open
#318 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
184
Forks
41
PR merge metrics
No merged PRs in 30d

Description

If network issues occur and connectivity is lost between the master and slaves, we retry network requests with exponential backoff. This is fine, but the backoff algorithm is not coordinated across threads -- each thread does its own backoff: all threads will retry after 0.1 seconds, then all will retry after 0.2 seconds, etc.

This occurred on a slave node but could affect the master as well.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named. Begin by locating the exponential-backoff retry path used for network requests between the master and slaves, then trace how concurrent threads schedule retries. Done means retries no longer synchronize into the same schedule across threads, with coverage for the affected slave-node behavior and any master path that shares it.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.