agentscope-ai / agentscope-ai/Trinity-RFT
`QueueWrapper.get_batch` in trainer hangs: how to debug?
- Ngôn ngữ chính
- Python
- Star
- 701
- Fork
- 79
- Merge trung bình
- 8 giờ 7 phút
- Pull request đã merge (30 ngày)
- 1
Mô tả
Hi!
Our trainer does not print any logs whatsoever and hangs indefinitely in `get_batch` (despite that explorer has already populated 100 megabytes in sqlite rollouts database).
How can we debug, understand the reason and fix the problem?
Thanks!
Stack traces from Ray dashboard:
```
Trainer process hung inside:
Process 423526: ray::Trainer.train
Python v3.12.11 (/usr/bin/python3.12)
Thread 423526 (idle): "MainThread"
get_objects (ray/_private/worker.py:934)
get (ray/_private/worker.py:2858)
wrapper (ray/_private/client_mode_hook.py:104)
auto_init_wrapper (ray/_private/auto_init_hook.py:22)
read (trinity/buffer/reader/queue_reader.py:32)
sample (trinity/algorithm/sample_strategy/sample_strategy.py:62)
train_step (trinity/trainer/trainer.py:70)
_resume_span (ray/util/tracing/tracing_helper.py:461)
train (trinity/trainer/trainer.py:52)
_resume_span (ray/util/tracing/tracing_helper.py:461)
actor_method_executor (ray/_private/function_manager.py:689)
main_loop (ray/_private/worker.py:974)
(ray/_private/workers/default_worker.py:321)
Thread 434749 (idle): "MsgRouterThr"
_read_packet_bytes (wandb/sdk/lib/sock_client.py:204)
read_server_response (wandb/sdk/lib/sock_client.py:231)
_read_message (wandb/sdk/interface/router_sock.py:27)
message_loop (wandb/sdk/interface/router.py:56)
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)
Thread 434859 (idle): "ChkStopThr"
wait (threading.py:359)
wait (threading.py:655)
_loop_check_status (wandb/sdk/wandb_run.py:265)
check_stop_status (wandb/sdk/wandb_run.py:307)
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)
Thread 434860 (idle): "NetStatThr"
wait (threading.py:359)
wait (threading.py:655)
wait_or (wandb/sdk/mailbox/response_handle.py:84)
wait_or (wandb/sdk/mailbox/mailbox_handle.py:118)
_loop_check_status (wandb/sdk/wandb_run.py:249)
check_network_status (wandb/sdk/wandb_run.py:282)
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)
Thread 434861 (idle): "IntMsgThr"
wait (threading.py:359)
wait (threading.py:655)
_loop_check_status (wandb/sdk/wandb_run.py:265)
check_internal_messages (wandb/sdk/wandb_run.py:334)
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)
QueueWrapper.get_batch:
Process 423524: ray::QueueWrapper
Python v3.12.11 (/usr/bin/python3.12)
Thread 423524 (idle): "MainThread"
main_loop (ray/_private/worker.py:974)
(ray/_private/workers/default_worker.py:321)
Thread 431786 (idle): "AsyncIO Thread: default"
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)
Thread 431897 (idle)
```
---
We frequently hit hangs and timeouts in various places :( Ray-based systems are so hard to debug :(
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.