agentscope-ai / agentscope-ai/Trinity-RFT
Log some sort of heartbeats to wandb (both from trainer and explorer): both at startup and in loops
- 主要言語
- Python
- スター
- 701
- フォーク
- 79
- 平均マージ
- 8時間 7分
- マージ済み PR(30日)
- 1
説明
We've experiences a problem with wandb logging while running a Trinity run:
- https://github.com/wandb/wandb/issues/10404
Seemingly we just ran the same Trinity config on a new machine set, but this for some reason led to no Trinity metrics being uploaded to Wandb: we waited for 20 hours, according to Ray Dashboard, Trinity was indeed having done some initial validation, and then some training steps - and nothing got uploaded to wandb except system metrics
I'm proposing to start logging some sort of heartbeat to wandb (and maybe allowing in config to force `commit=True`), so that there are some custom bits logged to Wandb almost immediately.
And if nothing gets logged/uploaded, it can immediately be assumed/confirmed problem with Wandb. Currently, if `eval_on_startup: true`, anything would get logged only in `_finish_explore_step(...)` and `_finish_eval_step(...)` which can take a while
Also, wandb has had a bug where `step=` was forcing it to ignore `commit=True` (not sure if this bug is still present):
- https://github.com/wandb/wandb/issues/3282#issuecomment-3234404317
コントリビューションガイド
評価
この issue はまだ評価されていません。