agentscope-ai / agentscope-ai/Trinity-RFT

Log some sort of heartbeats to wandb (both from trainer and explorer): both at startup and in loops

Aperta
#238 6 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
701
Fork
79
Merge medio
8h 7m
PR unite (30g)
1

Descrizione

We've experiences a problem with wandb logging while running a Trinity run:
- https://github.com/wandb/wandb/issues/10404

Seemingly we just ran the same Trinity config on a new machine set, but this for some reason led to no Trinity metrics being uploaded to Wandb: we waited for 20 hours, according to Ray Dashboard, Trinity was indeed having done some initial validation, and then some training steps - and nothing got uploaded to wandb except system metrics

I'm proposing to start logging some sort of heartbeat to wandb (and maybe allowing in config to force `commit=True`), so that there are some custom bits logged to Wandb almost immediately.

And if nothing gets logged/uploaded, it can immediately be assumed/confirmed problem with Wandb. Currently, if `eval_on_startup: true`, anything would get logged only in `_finish_explore_step(...)` and `_finish_eval_step(...)` which can take a while

Also, wandb has had a bug where `step=` was forcing it to ignore `commit=True` (not sure if this bug is still present):
- https://github.com/wandb/wandb/issues/3282#issuecomment-3234404317

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.