agentscope-ai / agentscope-ai/Trinity-RFT

`QueueWrapper.get_batch` in trainer hangs: how to debug?

Abierto
#182 6 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
701
Forks
79
Merge medio
8 h 7 min
PR fusionados (30 d)
1

Descripción

Hi!

Our trainer does not print any logs whatsoever and hangs indefinitely in `get_batch` (despite that explorer has already populated 100 megabytes in sqlite rollouts database).

How can we debug, understand the reason and fix the problem?

Thanks!

Image

Stack traces from Ray dashboard:
```
Trainer process hung inside:

Process 423526: ray::Trainer.train
Python v3.12.11 (/usr/bin/python3.12)

Thread 423526 (idle): "MainThread"
get_objects (ray/_private/worker.py:934)
get (ray/_private/worker.py:2858)
wrapper (ray/_private/client_mode_hook.py:104)
auto_init_wrapper (ray/_private/auto_init_hook.py:22)
read (trinity/buffer/reader/queue_reader.py:32)
sample (trinity/algorithm/sample_strategy/sample_strategy.py:62)
train_step (trinity/trainer/trainer.py:70)
_resume_span (ray/util/tracing/tracing_helper.py:461)
train (trinity/trainer/trainer.py:52)
_resume_span (ray/util/tracing/tracing_helper.py:461)
actor_method_executor (ray/_private/function_manager.py:689)
main_loop (ray/_private/worker.py:974)
(ray/_private/workers/default_worker.py:321)
Thread 434749 (idle): "MsgRouterThr"
_read_packet_bytes (wandb/sdk/lib/sock_client.py:204)
read_server_response (wandb/sdk/lib/sock_client.py:231)
_read_message (wandb/sdk/interface/router_sock.py:27)
message_loop (wandb/sdk/interface/router.py:56)
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)
Thread 434859 (idle): "ChkStopThr"
wait (threading.py:359)
wait (threading.py:655)
_loop_check_status (wandb/sdk/wandb_run.py:265)
check_stop_status (wandb/sdk/wandb_run.py:307)
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)
Thread 434860 (idle): "NetStatThr"
wait (threading.py:359)
wait (threading.py:655)
wait_or (wandb/sdk/mailbox/response_handle.py:84)
wait_or (wandb/sdk/mailbox/mailbox_handle.py:118)
_loop_check_status (wandb/sdk/wandb_run.py:249)
check_network_status (wandb/sdk/wandb_run.py:282)
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)
Thread 434861 (idle): "IntMsgThr"
wait (threading.py:359)
wait (threading.py:655)
_loop_check_status (wandb/sdk/wandb_run.py:265)
check_internal_messages (wandb/sdk/wandb_run.py:334)
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)

QueueWrapper.get_batch:

Process 423524: ray::QueueWrapper
Python v3.12.11 (/usr/bin/python3.12)

Thread 423524 (idle): "MainThread"
main_loop (ray/_private/worker.py:974)
(ray/_private/workers/default_worker.py:321)
Thread 431786 (idle): "AsyncIO Thread: default"
run (threading.py:1012)
_bootstrap_inner (threading.py:1075)
_bootstrap (threading.py:1032)
Thread 431897 (idle)
```

---

We frequently hit hangs and timeouts in various places :( Ray-based systems are so hard to debug :(

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.