agentscope-ai / agentscope-ai/Trinity-RFT

`QueueWrapper.get_batch` in trainer hangs: how to debug?

Open
#182 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
701
Forks
79
Avg merge
8h 7m
Merged PRs (30d)
1

Description

Hi!

Our trainer does not print any logs whatsoever and hangs indefinitely in `get_batch` (despite that explorer has already populated 100 megabytes in sqlite rollouts database).

How can we debug, understand the reason and fix the problem?

Thanks!

Image

Stack traces from Ray dashboard:
```
Trainer process hung inside:

Process 423526: ray::Trainer.train
Python v3.12.11 (/usr/bin/python3.12)

Thread 423526 (idle): "MainThread"
get_objects (ray/_private/worker.py:934)
get (ray/_private/worker.py:2858)
wrapper (ray/_private/client_mode_hook.py:104)
auto_init_wrapper (ray/_private/auto_init_hook.py:22)
read (trinity/buffer/reader/queue_reader.py:32)
sample (trinity/algorithm/sample_strategy/sample_strategy.py:62)
train_step (trinity/trainer/trainer.py:70)
_resume_span (ray/util/tracing/tracing_helper.py:461)
train (trinity/trainer/trainer.py:52)
_resume_span (ray/util/tracing/tracing_helper.py:461)
actor_method_executor (ray/_private/function_manager.py:689)
main_loop (ray/_private/worker.py:974)
(ray/_private/workers/default_worker.py:321)
Thread 434749 (idle): "MsgRouterThr"
_read_packet_bytes (wandb/sdk/lib/sock_client.py:204)
read_server_response (wandb/sdk/lib/sock_client.py:231)
_read_message (wandb/sdk/interface/router_sock.py:27)
message_loop (wandb/sdk/interface/router.py:56)
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)
Thread 434859 (idle): "ChkStopThr"
wait (threading.py:359)
wait (threading.py:655)
_loop_check_status (wandb/sdk/wandb_run.py:265)
check_stop_status (wandb/sdk/wandb_run.py:307)
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)
Thread 434860 (idle): "NetStatThr"
wait (threading.py:359)
wait (threading.py:655)
wait_or (wandb/sdk/mailbox/response_handle.py:84)
wait_or (wandb/sdk/mailbox/mailbox_handle.py:118)
_loop_check_status (wandb/sdk/wandb_run.py:249)
check_network_status (wandb/sdk/wandb_run.py:282)
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)
Thread 434861 (idle): "IntMsgThr"
wait (threading.py:359)
wait (threading.py:655)
_loop_check_status (wandb/sdk/wandb_run.py:265)
check_internal_messages (wandb/sdk/wandb_run.py:334)
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)

QueueWrapper.get_batch:

Process 423524: ray::QueueWrapper
Python v3.12.11 (/usr/bin/python3.12)

Thread 423524 (idle): "MainThread"
main_loop (ray/_private/worker.py:974)
(ray/_private/workers/default_worker.py:321)
Thread 431786 (idle): "AsyncIO Thread: default"
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)
Thread 431897 (idle)
```

---

We frequently hit hangs and timeouts in various places :( Ray-based systems are so hard to debug :(

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.