ServerlessBackend creates a new W&B run per training step, causing "Run finished" alerts to fire on every step
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 10.8k
- Forks
- 989
- Avg merge
- 6h 29m
- Merged PRs (30d)
- 85
Description
Hi, I ran into an issue while training with ServerlessBackend.
Problem
When using ServerlessBackend, each call to backend.train() creates a separate W&B run. This means that if "Run
finished" Slack notifications are enabled in W&B User Settings, a notification is sent on every training step — 50
steps means 50 Slack notifications, which is extremely noisy in practice.
Reproduction
for step in range(NUM_STEPS):
finished = await art.gather_trajectory_groups(groups)
result = await backend.train(model, finished, learning_rate=1e-5)
# ↑ Each call creates and finishes a separate W&B run
Expected behavior
One of the following would address this:
- Consolidate the entire training session (
model.register()throughbackend.close()) into a single W&B run - Handle per-step runs in a way that does not trigger W&B alerts (e.g. via
wandb.Settings) - Provide a user-configurable option to control this behavior
Workaround
Currently the only option is to disable "Run finished" notifications globally in W&B User Settings, but this silences
notifications for all projects — not ideal. W&B does not currently support per-project alert configuration.
Thanks for the great library!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing ServerlessBackend from model.register() through backend.train() and backend.close(), and inspect how each training call creates its W&B run. Reproduce the loop in the issue and evaluate the existing wandb.Settings option; done means a training session no longer produces an unwanted Run finished alert for every step.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100