agentscope-ai / agentscope-ai/Trinity-RFT

`QueueWrapper.get_batch` in trainer hangs: how to debug?

Đang mở
#182 6 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
701
Fork
79
Merge trung bình
8 giờ 7 phút
Pull request đã merge (30 ngày)
1

Mô tả

Hi!

Our trainer does not print any logs whatsoever and hangs indefinitely in `get_batch` (despite that explorer has already populated 100 megabytes in sqlite rollouts database).

How can we debug, understand the reason and fix the problem?

Thanks!

Image

Stack traces from Ray dashboard:
```
Trainer process hung inside:

Process 423526: ray::Trainer.train
Python v3.12.11 (/usr/bin/python3.12)

Thread 423526 (idle): "MainThread"
get_objects (ray/_private/worker.py:934)
get (ray/_private/worker.py:2858)
wrapper (ray/_private/client_mode_hook.py:104)
auto_init_wrapper (ray/_private/auto_init_hook.py:22)
read (trinity/buffer/reader/queue_reader.py:32)
sample (trinity/algorithm/sample_strategy/sample_strategy.py:62)
train_step (trinity/trainer/trainer.py:70)
_resume_span (ray/util/tracing/tracing_helper.py:461)
train (trinity/trainer/trainer.py:52)
_resume_span (ray/util/tracing/tracing_helper.py:461)
actor_method_executor (ray/_private/function_manager.py:689)
main_loop (ray/_private/worker.py:974)
(ray/_private/workers/default_worker.py:321)
Thread 434749 (idle): "MsgRouterThr"
_read_packet_bytes (wandb/sdk/lib/sock_client.py:204)
read_server_response (wandb/sdk/lib/sock_client.py:231)
_read_message (wandb/sdk/interface/router_sock.py:27)
message_loop (wandb/sdk/interface/router.py:56)
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)
Thread 434859 (idle): "ChkStopThr"
wait (threading.py:359)
wait (threading.py:655)
_loop_check_status (wandb/sdk/wandb_run.py:265)
check_stop_status (wandb/sdk/wandb_run.py:307)
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)
Thread 434860 (idle): "NetStatThr"
wait (threading.py:359)
wait (threading.py:655)
wait_or (wandb/sdk/mailbox/response_handle.py:84)
wait_or (wandb/sdk/mailbox/mailbox_handle.py:118)
_loop_check_status (wandb/sdk/wandb_run.py:249)
check_network_status (wandb/sdk/wandb_run.py:282)
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)
Thread 434861 (idle): "IntMsgThr"
wait (threading.py:359)
wait (threading.py:655)
_loop_check_status (wandb/sdk/wandb_run.py:265)
check_internal_messages (wandb/sdk/wandb_run.py:334)
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)

QueueWrapper.get_batch:

Process 423524: ray::QueueWrapper
Python v3.12.11 (/usr/bin/python3.12)

Thread 423524 (idle): "MainThread"
main_loop (ray/_private/worker.py:974)
(ray/_private/workers/default_worker.py:321)
Thread 431786 (idle): "AsyncIO Thread: default"
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)
Thread 431897 (idle)
```

---

We frequently hit hangs and timeouts in various places :( Ray-based systems are so hard to debug :(

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.