Lightning-AI / Lightning-AI/pytorch-lightning
Simple CloudWatch logger
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Description & Motivation
I'm using pytorch lightning and and torchmetrics for training. It works great on my local device and on AzureML - the tensorboard/mlflow loggers implemented in lightning do the job very well.
Recently I started training models on Sagemaker, and I don't see an easy/clean way to log metrics and track experiments in CloudWatch.
### Pitch
Recently I started training models on Sagemaker. I'd like to be able to track the training metrics while training directly in CloudWatch. To enable this, I've added `print()` to my code, which is ugly:
```
# Example of a training_step that's agnostic to the way the model is trained
def training_step(self, batch, batch_idx):
images, labels = batch
outputs = self(images)
# Loss
loss_value = self.loss(outputs, labels)
self.log('train_loss', loss_value, on_step=True, on_epoch=True)
# From outputs, get the class with the highest probability
predictions = torch.argmax(outputs, dim=1)
# Log metrics
self.train_accuracy(predictions, labels)
self.log('train_accuracy', self.train_accuracy, on_step=False, on_epoch=True)
# Save training_step_outputs -> added for CloudWatch support
self.training_step_outputs.append((loss_value, predictions, labels))
return loss_value
# For logging to CloudWatch, I need to write the logs manually
def on_train_epoch_end(self):
training_step_outputs = self.training_step_outputs
#... calculate metrics ...
print(f"train_loss:{train_loss};")
print(f"test_acc: {test_acc};")
self.training_step_outputs.clear() # free memory
```
In sagemaker related code, I specify the metrics regexes to match the print statements:
```
estimator = PyTorch(
# Some standard code
# ...
metric_definitions=[
{'Name': 'train_loss', 'Regex': 'train_loss:(.*?);'},
{'Name': 'train_accuracy', 'Regex': 'train_acc:(.*?);'},
],
enable_sagemaker_metrics=True,
)
```
I'd like to propose an implementation of a CloudWatch logger. It would:
- log every metric as `print(f'{metric_name}:{metric_value}')
Are there any suggestions how logging to CloudWatch could be more easily supported?
If this proposal makes sense to the commynity, I can implement the logger and create a PR.
### Alternatives
Add `print` statements to the `on_train_epoch_end`, as mentioned above.
### Additional context
_No response_
cc @borda @awaelchli @Blaizzy
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing TensorBoard and MLflow logger implementations mentioned in the issue, then compare their metric lifecycle with the example training_step and on_train_epoch_end hooks. The issue provides no repository files or tests; done would require an agreed CloudWatch logger scope and metrics that work with the shown SageMaker metric_definitions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- cloud, machine-learning, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100