Lightning-AI / Lightning-AI/pytorch-lightning
Logging metrics to mlflow in a `validation_step` at a lower frequency than once per epoch results in duplicate logs
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
Description & Motivation
Logging metrics to mlflow in a validation_step at a lower frequency than once per epoch results in duplicate logs.
I am experimenting with the usage of pytorch lightning with a limited sample size, requiring many training epochs ~10_000. I use mlflow to manage my experiments. Mostly, I log with self.log(name, number) within validation_step, which gets called every 50 epochs (via the check_val_every_n_epoch argument to pl.Trainer). However, it appears that for each call to self.log(...) in validation_step, 50 duplicate lines are written to the corresponding metric file. This results in large metric text files and thus large experiments, which mlflow appears to struggle with (see, e.g., here).
Expand for output
(base) [lmalte@eu-login-12 metrics]$ tail -n 105 val_l2_loss
1679859669342 7.820003986358643 395
1679859670737 7.820003986358643 396
1679859672141 7.820003986358643 397
1679859673531 7.820003986358643 398
1679859681443 7.73037576675415 399
1679859682855 7.73037576675415 400
1679859684166 7.73037576675415 401
1679859685567 7.73037576675415 402
1679859686975 7.73037576675415 403
1679859688384 7.73037576675415 404
1679859689881 7.73037576675415 405
1679859691312 7.73037576675415 406
1679859692611 7.73037576675415 407
1679859694130 7.73037576675415 408
1679859695447 7.73037576675415 409
1679859696951 7.73037576675415 410
1679859698289 7.73037576675415 411
1679859699774 7.73037576675415 412
1679859701152 7.73037576675415 413
1679859702552 7.73037576675415 414
1679859703970 7.73037576675415 415
1679859705387 7.73037576675415 416
1679859706886 7.73037576675415 417
1679859708316 7.73037576675415 418
1679859709634 7.73037576675415 419
1679859711155 7.73037576675415 420
1679859712479 7.73037576675415 421
1679859714009 7.73037576675415 422
1679859715338 7.73037576675415 423
1679859716842 7.73037576675415 424
1679859718168 7.73037576675415 425
1679859719561 7.73037576675415 426
1679859720963 7.73037576675415 427
1679859722377 7.73037576675415 428
1679859723913 7.73037576675415 429
1679859725324 7.73037576675415 430
1679859726639 7.73037576675415 431
1679859728142 7.73037576675415 432
1679859729469 7.73037576675415 433
1679859730971 7.73037576675415 434
1679859732299 7.73037576675415 435
1679859733795 7.73037576675415 436
1679859735127 7.73037576675415 437
1679859736549 7.73037576675415 438
1679859737973 7.73037576675415 439
1679859739379 7.73037576675415 440
1679859740858 7.73037576675415 441
1679859742283 7.73037576675415 442
1679859743599 7.73037576675415 443
1679859745118 7.73037576675415 444
1679859746425 7.73037576675415 445
1679859747920 7.73037576675415 446
1679859749252 7.73037576675415 447
1679859750757 7.73037576675415 448
1679859758637 7.764103889465332 449
1679859759893 7.764103889465332 450
1679859761319 7.764103889465332 451
1679859762726 7.764103889465332 452
1679859764151 7.764103889465332 453
1679859765572 7.764103889465332 454
1679859766976 7.764103889465332 455
1679859768390 7.764103889465332 456
1679859769777 7.764103889465332 457
1679859771199 7.764103889465332 458
1679859772597 7.764103889465332 459
1679859774017 7.764103889465332 460
1679859775440 7.764103889465332 461
1679859776868 7.764103889465332 462
1679859778304 7.764103889465332 463
1679859779709 7.764103889465332 464
1679859781139 7.764103889465332 465
1679859782563 7.764103889465332 466
1679859783974 7.764103889465332 467
1679859785387 7.764103889465332 468
1679859786791 7.764103889465332 469
1679859788208 7.764103889465332 470
1679859789613 7.764103889465332 471
1679859791025 7.764103889465332 472
1679859792438 7.764103889465332 473
1679859793838 7.764103889465332 474
1679859795270 7.764103889465332 475
1679859796662 7.764103889465332 476
1679859798078 7.764103889465332 477
1679859799475 7.764103889465332 478
1679859800880 7.764103889465332 479
1679859802294 7.764103889465332 480
1679859803687 7.764103889465332 481
1679859805110 7.764103889465332 482
1679859806507 7.764103889465332 483
1679859807922 7.764103889465332 484
1679859809323 7.764103889465332 485
1679859810733 7.764103889465332 486
1679859812140 7.764103889465332 487
1679859813543 7.764103889465332 488
1679859814971 7.764103889465332 489
1679859816376 7.764103889465332 490
1679859817782 7.764103889465332 491
1679859819199 7.764103889465332 492
1679859820597 7.764103889465332 493
1679859822017 7.764103889465332 494
1679859823407 7.764103889465332 495
1679859824819 7.764103889465332 496
1679859826232 7.764103889465332 497
1679859827633 7.764103889465332 498
1679859835503 7.853664875030518 499
Pitch
Allow to only write only one line per call to self.log in validation_step to the mlflow logs.
Alternatives
Don't use pytorch-lightning for mlflog (auto-)logging.
Additional context
No response
cc @borda
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the validation_step logging path, including self.log and the check_val_every_n_epoch behavior described in the issue. Reproduce the MLflow output with validation every 50 epochs and verify that each self.log call produces one metric line rather than repeated lines.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100