delay_service 已消费的延迟任务只在内存线程 sleep——进程重启即丢唤醒,执行永久 suspended
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2h 28m
- Merged PRs (30d)
- 5
Description
现象
delay 节点的执行链路存在一个可靠性缺口:worker 挂起时把任务 RPUSH 到 plaita:delay:queue,delay_service BLPOP 消费后在内存线程里 sleep 到触发时刻(base_service.py ThreadPoolExecutor + delay_service.py 的 trigger 循环)。如果 delay_service 在「已消费、未触发」窗口内重启/被杀,任务丢失且无任何补偿——对应执行永久停留在 suspended,只能人工 resume。
证据
- 任务里已经带了
planned_trigger_timestamp(delay_service 构造 service_config 时写入),但没有任何启动扫描/补偿逻辑读它 - 对比:schedule_service 有 next_run_at 落 Redis + 每秒扫描的自愈设计;delay 队列的任务一旦出队就只活在内存里
- E2E 混沌演练中确认:ext-services 容器在 delay 执行中途被杀 → 该执行无法恢复(这是设计缺口,不是回归,故未做成断言)
建议
延迟任务出队后把 {execution_id, planned_trigger_timestamp, event_type, task_config} 落 Redis(如 plaita:delay:pending hash),触发后删除;delay_service 启动时扫描该结构,对已到期的任务直接补触发、未到期的重新调度。这样重启窗口内的延迟任务可恢复。
影响
多容器部署(docker-compose / k8s)下 ext-services 的例行重启/滚动更新都会触发此缺口。当前 5 容器 E2E 拓扑里它被 chaos 脚本刻意绕开(不动 ext-services),但生产上绕不开。
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read base_service.py and delay_service.py to trace how delay tasks move from the Redis queue into the in-memory trigger loop, then compare the recovery approach in schedule_service. Implement persistent pending-task tracking and startup recovery for due and future tasks, and verify that tasks survive a delay_service restart without remaining suspended.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100