jeffkit / jeffkit/plaita

delay_service 已消费的延迟任务只在内存线程 sleep——进程重启即丢唤醒,执行永久 suspended

Open
#11 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
0
Avg merge
2h 28m
Merged PRs (30d)
5

Description

现象

delay 节点的执行链路存在一个可靠性缺口:worker 挂起时把任务 RPUSH 到 plaita:delay:queue,delay_service BLPOP 消费后在内存线程里 sleep 到触发时刻base_service.py ThreadPoolExecutor + delay_service.py 的 trigger 循环)。如果 delay_service 在「已消费、未触发」窗口内重启/被杀,任务丢失且无任何补偿——对应执行永久停留在 suspended,只能人工 resume。

证据

  • 任务里已经带了 planned_trigger_timestamp(delay_service 构造 service_config 时写入),但没有任何启动扫描/补偿逻辑读它
  • 对比:schedule_service 有 next_run_at 落 Redis + 每秒扫描的自愈设计;delay 队列的任务一旦出队就只活在内存里
  • E2E 混沌演练中确认:ext-services 容器在 delay 执行中途被杀 → 该执行无法恢复(这是设计缺口,不是回归,故未做成断言)

建议

延迟任务出队后把 {execution_id, planned_trigger_timestamp, event_type, task_config} 落 Redis(如 plaita:delay:pending hash),触发后删除;delay_service 启动时扫描该结构,对已到期的任务直接补触发、未到期的重新调度。这样重启窗口内的延迟任务可恢复。

影响

多容器部署(docker-compose / k8s)下 ext-services 的例行重启/滚动更新都会触发此缺口。当前 5 容器 E2E 拓扑里它被 chaos 脚本刻意绕开(不动 ext-services),但生产上绕不开。

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read base_service.py and delay_service.py to trace how delay tasks move from the Redis queue into the in-memory trigger loop, then compare the recovery approach in schedule_service. Implement persistent pending-task tracking and startup recovery for due and future tasks, and verify that tasks survive a delay_service restart without remaining suspended.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.