agentscope-ai / agentscope-ai/Trinity-RFT

Log some sort of heartbeats to wandb (both from trainer and explorer): both at startup and in loops

未关闭
#238 6 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
701
派生
79
平均合并
8 小时 7 分钟
30 天内合并 PR
1

描述

We've experiences a problem with wandb logging while running a Trinity run:
- https://github.com/wandb/wandb/issues/10404

Seemingly we just ran the same Trinity config on a new machine set, but this for some reason led to no Trinity metrics being uploaded to Wandb: we waited for 20 hours, according to Ray Dashboard, Trinity was indeed having done some initial validation, and then some training steps - and nothing got uploaded to wandb except system metrics

I'm proposing to start logging some sort of heartbeat to wandb (and maybe allowing in config to force `commit=True`), so that there are some custom bits logged to Wandb almost immediately.

And if nothing gets logged/uploaded, it can immediately be assumed/confirmed problem with Wandb. Currently, if `eval_on_startup: true`, anything would get logged only in `_finish_explore_step(...)` and `_finish_eval_step(...)` which can take a while

Also, wandb has had a bug where `step=` was forcing it to ignore `commit=True` (not sure if this bug is still present):
- https://github.com/wandb/wandb/issues/3282#issuecomment-3234404317

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。