codingjoe / codingjoe/threadmill
ExponentialBackoff overflows timedelta above attempt 46
- 主要语言
- Python
- 星标
- 12
- 派生
- 1
- 平均合并
- 1 天 1 小时
- 30 天内合并 PR
- 10
描述
`ExponentialBackoff.__call__` computes
```python
delay = min(self.base_delay * (self.factor**context.attempt), self.max_delay)
```
so the product is evaluated before `min()` clamps it. With the default shape (base delay 1 s, factor 2.0, max delay 1 h) the product exceeds the `timedelta` limit at attempt 47:
```
OverflowError: days=1628906115; must have magnitude <= 999999999
```
### Repro (threadmill 0.7.1)
```python
import datetime
from types import SimpleNamespace
from threadmill.retry import ExponentialBackoff
policy = ExponentialBackoff(
base_delay=datetime.timedelta(seconds=1),
max_delay=datetime.timedelta(hours=1),
factor=2.0,
max_retries=720,
)
def context(attempt):
error = SimpleNamespace(exception_class=ValueError)
return SimpleNamespace(attempt=attempt, task_result=SimpleNamespace(errors=[error]))
for attempt in range(1, 60):
try:
print(attempt, policy(context(attempt)))
except Exception as exc:
print(attempt, type(exc).__name__, exc)
break
```
Attempts 1 to 46 return a delay (2 s doubling to the 1 h cap at attempt 12), attempt 47 raises.
### Why it matters
`Executor.retry_delay` catches the exception, logs `Retry callback failed`, and returns `None`, so the backend acknowledges the result and the retry chain ends. The task looks like it exhausted its policy, but `max_retries` was never reached: a policy of 720 attempts really stops after 46 retries.
### Suggested fix
Clamp in seconds before building the `timedelta`, and derive the capped attempt count from `max_delay` so the exponential is never evaluated past the cap:
```python
seconds = min(
self.base_delay.total_seconds() * self.factor**context.attempt,
self.max_delay.total_seconds(),
)
return datetime.timedelta(seconds=seconds)
```
A plain `min()` on the two `timedelta` values does not help on its own, because the product still overflows before the comparison.
Found while bounding the spam scan retry budget in codingjoe/relay#230.
贡献指南
调研方向
从 threadmill.retry.ExponentialBackoff.__call__ 开始,检查在应用 min() 之前其延迟是如何计算的。使用提供的尝试循环重现该问题,然后检查 Executor.retry_delay,以确认 callback 在尝试次数较高时不再失败。当直到 max_retries 的尝试(包括 max_retries)都保持限制在 max_delay,而不是以 overflow 结束时,即表示完成。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- backend
- Issue 类型
- 缺陷
- 难度
- 2/5
- 预计耗时
- 1-3 小时
- 活跃度
- 活跃
- 描述清晰度
- 描述清楚
- 新手友好度
- 78/100