Lightning-AI / Lightning-AI/pytorch-lightning

Formatting for grouped metrics.

Open
#10,097 10 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

feature logger priority: 1
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

🚀 Feature

Motivation

With Lightning, I use Tensorboard and ClearML, which allow you to group metrics in one tab. The "/" is used for grouping. In the screenshots from tb and ClearML you can see how they are grouped now.
Screenshot 2021-10-23 at 22 14 43
Screenshot 2021-10-23 at 22 22 04
Screenshot 2021-10-23 at 22 23 46

Pitch

  1. The first is that when you select on_step=True and on_epoch=True, a postfix is appended to the end and the metrics by epoch and step are grouped into one in ClearML. I would like to split this group into two: loss_step/<train or valid> and loss_epoch/<train or valid>. Could the name parameter in self.log be formatted? What do you think? This would be something like name="loss_{interval}/valid".

  2. The second problem also involves formatting. When you pass a grouped metric to filename in ModelCheckpoint, Lightning will think "/" means to create a folder 🙂 See screenshot. Would it be possible to add an escaping for slash and replace it with a lower underscore?

Screenshot 2021-10-23 at 22 24 12 Screenshot 2021-10-23 at 22 24 42
Alternatives
Additional context

Similar problem, but with W&B in Lightning-Flash https://github.com/PyTorchLightning/lightning-flash/issues/818

If you enjoy Lightning, check out our other projects! ⚡

  • Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

  • Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, finetuning and solving problems with deep learning

  • Bolts: Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

  • Lightning Transformers: Flexible interface for high performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the self.log entry point and ModelCheckpoint filename handling mentioned in the issue. Trace how interval suffixes and slash-separated metric names are currently produced, then add focused tests showing the requested grouping and safe checkpoint filenames are complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.