Lightning-AI / Lightning-AI/pytorch-lightning

WandbLogger does not log losses with self.log() in LightningModule

Open
#20,600 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug logger: wandb ver: 2.4.x
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

Hello everyone,

I have been using lightning to train my model, and relying on `WandLogger` to log metrics and losses.
As a side information, I am running the code on nodes from a cluster that disables internet access for computing nodes. Thus it requires offline wandb sync after/during the training job.

I have encountered the following behavior,
- System logs are all logged on wandb and I can access them on wandb.ai without any trouble.
- However, logging the loss values on wandb does not work using `self.log()` inside my `LightningModule`. The values are correctly printed during training, but I cannot access them on wandb.ai after sync.
- When calling `self.logger.experiment.log()`, the loss values are correctly logged and accessible on wandb.ai.

### What version are you seeing the problem on?

v2.4

### How to reproduce the bug

The main file:
```python
...
wandb_logger = WandbLogger(
project="ProjectName",
offline=True,
config=config,)
...
trainer = pl.Trainer(
max_epochs=config['training']['num_epochs'],
accumulate_grad_batches=config['training']['gradient_accumulation_steps'],
callbacks=callbacks,
devices=int(os.environ['SLURM_GPUS_ON_NODE']),
accelerator=device_type,
strategy=strategy,
logger=wandb_logger,
profiler=profiler,
enable_model_summary=True,
num_nodes=int(os.environ['SLURM_NNODES']),
detect_anomaly=config['anomaly_run'],
fast_dev_run=config['fast_dev_run'],
)
trainer.fit(pl_model,
dataset,
ckpt_path=load_pth)

```

The`LightningModule`:
```python
import lightning as pl

class MyClass(pl.LightningModule):
def __init__(self,
model: nn.Module,
config: Dict):

...

def training_step(self, batch, batch_idx):
...
asr_loss = ...
em_loss = ...
total_loss = asr_loss + em_loss

self.log("Train/ASR_step_loss",
asr_loss.item(),
prog_bar=True,
on_step=True,
on_epoch=True,
sync_dist=True,
logger=True,)
self.log("Train/total_loss",
total_loss.item(),
prog_bar=True,
on_step=True,
on_epoch=True,
sync_dist=True,
logger=True,)
self.log("Train/EM_step_loss",
loss_em.item(),
prog_bar=True,
on_step=True,
on_epoch=True,
sync_dist=True,
logger=True,)

# without this code, loss are not logged on wandb
self.logger.experiment.log(
{"Train/ASR_step_loss":
asr_loss.item()},
)
self.logger.experiment.log(
{"Train/em_loss":
em_loss.item()},
)
self.logger.experiment.log(
{"Train/total_loss":
total_loss.item()},
)

return total_loss
```

### Error messages and logs

As a side note, the wandb logs provide the following:
```
2025-02-21 11:02:09,547 ERROR MainThread:3724374 [wandb_run.py:_redirect():2320] Failed to redirect.
Traceback (most recent call last):
File "path/to/.local/lib/python3.11/site-packages/wandb/sdk/wandb_run.py", line 2313, in _redirect
out_redir.install()
File "path/to/.local/lib/python3.11/site-packages/wandb/sdk/lib/redirect.py", line 654, in install
stream.write = write
^^^^^^^^^^^^
AttributeError: 'stream.Stream' object attribute 'write' is read-only
2025-02-21 11:02:09,556 INFO MainThread:3724374 [wandb_init.py:init():1037] run started, returning control to user process
```

### Environment

Current environment

* CUDA:
- GPU:
- Tesla V100-SXM2-16GB
- available: True
- version: 12.2
* Lightning:
- efficientnet-pytorch: 0.7.1
- idr-torch: 2.2.0
- lightning: 2.4.0
- lightning-utilities: 0.11.8
- pytorch-fast-transformers: 0.4.0
- pytorch-ignite: 0.5.1
- pytorch-lightning: 2.4.0
- pytorch-msssim: 1.0.0
- pytorch-pfn-extras: 0.7.7
- pytorchvideo: 0.1.5
- segmentation-models-pytorch: 0.3.4
- torch: 2.4.0
- torch-cluster: 1.6.3
- torch-ema: 0.3
- torch-geometric: 2.6.1
- torch-scatter: 2.1.2
- torch-sparse: 0.6.18
- torch-spline-conv: 1.2.2
- torch-tb-profiler: 0.4.3
- torchao: 0.6.1
- torchaudio: 2.4.0a0+69d4077
- torchdata: 0.8.0
- torchensemble: 0.2.0
- torchinfo: 1.8.0
- torchio: 0.20.1
- torchmetrics: 1.4.3
- torchtune: 0.3.1
- torchvision: 0.19.0a0
- torchviz: 0.0.2
* Packages:
- absl-py: 2.1.0
- accelerate: 1.0.1
- addict: 2.4.0
- aenum: 3.1.15
- affine: 2.4.0
- aim: 3.25.0
- aim-ui: 3.25.0
- aimrecords: 0.0.7
- aimrocks: 0.5.2
- aiofiles: 23.2.1
- aiohappyeyeballs: 2.4.3
- aiohttp: 3.10.10
- aiohttp-cors: 0.7.0
- aiosignal: 1.3.1
- alabaster: 0.7.16
- albucore: 0.0.18
- albumentations: 1.4.18
- alembic: 1.13.3
- aniso8601: 9.0.1
- annotated-types: 0.7.0
- antlr4-python3-runtime: 4.9.3
- anyio: 4.2.0
- apex: 0.1
- appdirs: 1.4.4
- argon2-cffi: 23.1.0
- argon2-cffi-bindings: 21.2.0
- arrow: 1.3.0
- asgiref: 3.8.1
- asn1crypto: 1.5.1
- assertpy: 1.1
- astroid: 3.3.5
- astropy: 6.1.4
- astropy-iers-data: 0.2024.10.14.0.32.55
- asttokens: 2.4.1
- astunparse: 1.6.3
- async-lru: 2.0.4
- asyncssh: 2.17.0
- atomicwrites: 1.4.1
- attrs: 24.2.0
- audioread: 3.0.1
- autocommand: 2.2.2
- autopep8: 2.0.4
- av: 13.1.0
- awscli: 1.35.8
- babel: 2.16.0
- backoff: 2.2.1
- backports.tarfile: 1.2.0
- base58: 2.0.1
- basicsr: 1.4.2
- bcrypt: 4.2.0
- beautifulsoup4: 4.12.3
- binaryornot: 0.4.4
- biopython: 1.84
- bitarray: 2.5.1
- bitsandbytes: 0.44.1
- bkcharts: 0.2
- black: 24.10.0
- bleach: 6.1.0
- blinker: 1.6.2
- blis: 0.7.11
- blobfile: 3.0.0
- bokeh: 3.6.0
- boto: 2.49.0
- boto3: 1.35.42
- botocore: 1.35.42
- bottleneck: 1.4.1
- braceexpand: 0.1.7
- branca: 0.8.0
- build: 1.2.2.post1
- cachetools: 5.5.0
- catalogue: 2.0.10
- ccimport: 0.4.4
- certifi: 2024.8.30
- cffi: 1.17.1
- chardet: 5.2.0
- charset-normalizer: 3.4.0
- chroma-hnswlib: 0.7.6
- chromadb: 0.5.15
- clarabel: 0.9.0
- click: 8.1.7
- click-plugins: 1.1.1
- cligj: 0.7.2
- clip: 0.2.0
- cloudpathlib: 0.19.0
- cloudpickle: 3.1.0
- cmake: 3.30.4
- colorama: 0.4.6
- coloredlogs: 15.0.1
- colorful: 0.5.6
- colorlog: 6.8.2
- comm: 0.2.2
- compressai: 1.2.6
- confection: 0.1.5
- contourpy: 1.3.0
- cookiecutter: 2.6.0
- cryptography: 43.0.1
- cumm-cu120: 0.4.11
- cupy-cuda12x: 13.3.0
- cvxpy: 1.5.3
- cvxpylayers: 0.1.6
- cxxfilt: 0.3.0
- cycler: 0.12.1
- cymem: 2.0.8
- cytoolz: 0.12.2
- dask: 2024.8.2
- dask-expr: 1.1.13
- dask-jobqueue: 0.8.2
- databricks-sdk: 0.35.0
- dataclasses: 0.6
- dataclasses-json: 0.6.7
- datasets: 2.12.0
- debugpy: 1.8.7
- decorator: 5.1.1
- deepspeed: 0.15.2
- defusedxml: 0.7.1
- deprecated: 1.2.13
- dgl: 2.1.0
- dglgo: 0.0.2
- diff-match-patch: 20230430
- diffcp: 1.0.23
- diffusers: 0.30.3
- dill: 0.3.6
- diskcache: 5.6.3
- distarray: 0.6.0
- distlib: 0.3.9
- distributed: 2024.8.2
- distro: 1.9.0
- dm-tree: 0.1.8
- docker: 7.1.0
- docker-pycreds: 0.4.0
- docopt: 0.6.2
- docstring-parser: 0.16
- docstring-to-markdown: 0.15
- docutils: 0.21.2
- durationpy: 0.9
- eagerpy: 0.30.0
- ecos: 2.0.14
- efficientnet-pytorch: 0.7.1
- einops: 0.8.0
- embreex: 2.17.7.post5
- entrypoints: 0.4
- eo-learn: 1.5.7
- eo-learn-core: 1.5.0
- eo-learn-coregistration: 1.5.0
- eo-learn-features: 1.5.0
- eo-learn-geometry: 1.5.0
- eo-learn-io: 1.5.0
- eo-learn-mask: 1.5.0
- eo-learn-ml-tools: 1.5.0
- eo-learn-visualization: 1.5.0
- evaluate: 0.4.3
- executing: 2.1.0
- expecttest: 0.2.1
- fabric: 2.6.0
- fastai: 2.7.17
- fastapi: 0.115.2
- fastargs: 1.2.0
- fastcore: 1.7.19
- fastdownload: 0.0.7
- fastjsonschema: 2.20.0
- fastkml: 0.12
- fastmri: 0.3.0
- fastprogress: 1.0.3
- fastrlock: 0.8.2
- ffcv: 1.0.2
- ffmpy: 0.4.0
- filelock: 3.13.1
- filetype: 1.2.0
- findiff: 0.10.2
- fiona: 1.10.1
- fire: 0.7.0
- flake8: 7.1.1
- flash-attn: 2.6.1
- flask: 3.0.3
- flatbuffers: 24.3.25
- folium: 0.17.0
- fonttools: 4.54.1
- foolbox: 3.3.4
- fqdn: 1.5.1
- frozenlist: 1.4.1
- fs: 2.4.16
- fs-s3fs: 1.1.1
- fsspec: 2023.6.0
- future: 1.0.0
- fvcore: 0.1.6
- gast: 0.5.3
- geographiclib: 2.0
- geopandas: 0.14.4
- gguf: 0.10.0
- gitdb: 4.0.11
- gitpython: 3.1.43
- glob2: 0.7
- global-land-mask: 1.0.0
- gmpy2: 2.1.2
- google-api-core: 2.21.0
- google-auth: 2.35.0
- googleapis-common-protos: 1.65.0
- gradio: 5.1.0
- gradio-client: 1.4.0
- graphene: 3.3
- graphql-core: 3.2.5
- graphql-relay: 3.2.0
- graphviz: 0.20.3
- greenlet: 3.1.1
- grpcio: 1.67.0
- gunicorn: 23.0.0
- guppy3: 3.1.4.post1
- h11: 0.14.0
- h5py: 3.12.1
- heapdict: 1.0.1
- hjson: 3.1.0
- horovod: 0.28.1
- httpcore: 1.0.6
- httptools: 0.6.4
- httpx: 0.27.2
- huggingface-hub: 0.26.0
- humanfriendly: 10.0
- humanize: 4.11.0
- hydra-core: 1.3.2
- hyperpyyaml: 1.2.2
- hypothesis: 6.115.2
- idna: 3.7
- idr-accelerate: 2.0.1
- idr-env-protect: 2.0.0
- idr-pypi: 1.0.0
- idr-torch: 2.2.0
- idris: 1.0.0
- imageio: 2.33.1
- imageio-ffmpeg: 0.5.1
- imagesize: 1.4.1
- imbalanced-learn: 0.12.4
- importlib-metadata: 8.4.0
- importlib-resources: 6.4.0
- imutils: 0.5.4
- inflect: 7.3.1
- inflection: 0.5.1
- iniconfig: 1.1.1
- interegular: 0.3.3
- intervaltree: 3.1.0
- invoke: 1.6.0
- iopath: 0.1.10
- ipykernel: 6.29.5
- ipyparallel: 8.8.0
- ipython: 8.28.0
- ipython-genutils: 0.2.0
- ipywidgets: 8.1.5
- isoduration: 20.11.0
- isort: 5.13.2
- itsdangerous: 2.2.0
- jaraco.classes: 3.4.0
- jaraco.collections: 5.1.0
- jaraco.context: 6.0.1
- jaraco.functools: 4.1.0
- jaraco.text: 3.12.1
- jedi: 0.19.1
- jeepney: 0.8.0
- jellyfish: 1.1.0
- jinja2: 3.1.4
- jiter: 0.6.1
- jmespath: 1.0.1
- joblib: 1.4.2
- json5: 0.9.25
- jsonpatch: 1.33
- jsonpointer: 3.0.0
- jsonschema: 4.23.0
- jsonschema-specifications: 2024.10.1
- jupyter: 1.1.1
- jupyter-client: 8.6.0
- jupyter-console: 6.6.3
- jupyter-contrib-core: 0.4.2
- jupyter-core: 5.7.2
- jupyter-events: 0.10.0
- jupyter-lsp: 2.2.5
- jupyter-nbextensions-configurator: 0.6.4
- jupyter-server: 2.14.1
- jupyter-server-terminals: 0.4.4
- jupyter-tensorboard: 0.2.0
- jupyterlab: 4.2.5
- jupyterlab-pygments: 0.1.2
- jupyterlab-server: 2.27.3
- jupyterlab-widgets: 3.0.13
- jupyterquiz: 2.6.3
- keyring: 25.4.1
- kiwisolver: 1.4.7
- kornia: 0.7.3
- kornia-rs: 0.1.5
- kubernetes: 31.0.0
- langchain: 0.3.3
- langchain-core: 0.3.12
- langchain-text-splitters: 0.3.0
- langcodes: 3.4.1
- langsmith: 0.1.136
- language-data: 1.2.0
- lapjv: 1.3.27
- lark: 1.2.2
- lazy-loader: 0.4
- librosa: 0.10.2.post1
- lief: 0.15.1
- lightgbm: 4.5.0
- lightning: 2.4.0
- lightning-utilities: 0.11.8
- linkify-it-py: 2.0.3
- lintrunner: 0.12.5
- littleutils: 0.2.4
- llvmlite: 0.43.0
- lm-format-enforcer: 0.10.6
- lmdb: 1.4.1
- locket: 1.0.0
- lpips: 0.1.4
- lxml: 5.2.1
- mako: 1.3.5
- manifold3d: 2.5.1
- mapbox-earcut: 1.0.2
- marisa-trie: 1.2.1
- markdown: 3.7
- markdown-it-py: 3.0.0
- markupsafe: 2.1.5
- marshmallow: 3.22.0
- matplotlib: 3.9.2
- matplotlib-inline: 0.1.7
- mccabe: 0.7.0
- mdit-py-plugins: 0.4.2
- mdurl: 0.1.2
- megatron-lm: 3.0.0
- memray: 1.14.0
- mistral-common: 1.4.4
- mistune: 3.0.2
- mizani: 0.11.4
- mlconfig: 0.2.2
- mlflow: 2.17.0
- mlflow-skinny: 2.17.0
- mmcv: 2.2.0
- mmengine: 0.10.5
- mmh3: 5.0.1
- mock: 4.0.3
- monai: 1.3.1
- monotonic: 1.6
- more-itertools: 10.5.0
- moses: 0.10.0
- mosestokenizer: 1.2.1
- mpi4py: 4.0.1
- mpmath: 1.3.0
- msgpack: 1.0.3
- msgspec: 0.18.6
- multidict: 6.1.0
- multipledispatch: 0.6.0
- multiprocess: 0.70.14
- munch: 4.0.0
- murmurhash: 1.0.10
- mypy-extensions: 1.0.0
- natsort: 8.4.0
- nbclient: 0.10.0
- nbconvert: 7.16.4
- nbformat: 5.10.4
- nest-asyncio: 1.6.0
- networkx: 3.2.1
- nh3: 0.2.18
- nibabel: 5.3.1
- ninja: 1.11.1.1
- nltk: 3.9.1
- notebook: 6.4.12
- notebook-shim: 0.2.4
- numba: 0.60.0
- numpy: 1.26.4
- numpydoc: 1.8.0
- nvidia-dali-cuda120: 1.42.0
- nvidia-dlprof: 0.17.0
- nvidia-ml-py: 12.560.30
- nvidia-nsys-cli: 2021.3.2.12
- nvidia-nvimgcodec-cu12: 0.3.0.5
- nvidia-pyprof: 3.10.0
- oauthlib: 3.2.2
- ogb: 1.3.6
- olefile: 0.46
- omegaconf: 2.3.0
- onnxruntime: 1.19.2
- openai: 1.51.2
- openai-whisper: 20240930
- opencensus: 0.11.4
- opencensus-context: 0.1.3
- opencv-python: 4.10.0.84
- opencv-python-headless: 4.10.0.84
- openfile: 0.0.7
- opentelemetry-api: 1.27.0
- opentelemetry-exporter-otlp-proto-common: 1.27.0
- opentelemetry-exporter-otlp-proto-grpc: 1.27.0
- opentelemetry-instrumentation: 0.48b0
- opentelemetry-instrumentation-asgi: 0.48b0
- opentelemetry-instrumentation-fastapi: 0.48b0
- opentelemetry-proto: 1.27.0
- opentelemetry-sdk: 1.27.0
- opentelemetry-semantic-conventions: 0.48b0
- opentelemetry-util-http: 0.48b0
- opt-einsum: 3.4.0
- optree: 0.13.0
- optuna: 4.0.0
- orjson: 3.10.7
- osqp: 0.6.7.post3
- outdated: 0.2.2
- outlines: 0.0.46
- overrides: 7.7.0
- packaging: 24.1
- pafit: 2.0.8
- pandas: 2.2.3
- pandocfilters: 1.5.1
- parameterized: 0.9.0
- paramiko: 2.8.1
- parso: 0.8.4
- partd: 1.4.1
- partial-json-parser: 0.2.1.1.post4
- pathlib2: 2.3.6
- pathspec: 0.12.1
- patsy: 0.5.6
- pccm: 0.4.16
- peft: 0.13.2
- pexpect: 4.9.0
- photutils: 2.0.1
- pickleshare: 0.7.5
- pillow: 10.4.0
- pip: 24.2
- pkginfo: 1.10.0
- platformdirs: 3.10.0
- plotbin: 3.1.7
- plotly: 5.24.1
- plotnine: 0.13.6
- pluggy: 1.5.0
- ply: 3.11
- plyfile: 1.1
- pooch: 1.8.2
- portalocker: 2.10.1
- posthog: 3.7.0
- potpourri3d: 1.1.0
- preshed: 3.0.9
- pretrainedmodels: 0.7.4
- prettytable: 3.14.0
- prometheus-client: 0.14.1
- prometheus-fastapi-instrumentator: 7.0.0
- prompt-toolkit: 3.0.48
- propcache: 0.2.0
- proto-plus: 1.24.0
- protobuf: 4.25.5
- psutil: 6.0.0
- ptyprocess: 0.7.0
- pure-eval: 0.2.3
- py-cpuinfo: 9.0.0
- py-spy: 0.3.14
- pyairports: 2.1.1
- pyarrow: 17.0.0
- pyasn1: 0.6.1
- pyasn1-modules: 0.4.1
- pybind11: 2.12.0
- pybind11-global: 2.12.0
- pycairo: 1.23.0
- pycodestyle: 2.12.1
- pycollada: 0.8
- pyconll: 3.2.0
- pycountry: 24.6.1
- pycparser: 2.22
- pycryptodomex: 3.21.0
- pycurl: 7.45.3
- pydantic: 2.9.2
- pydantic-core: 2.23.4
- pydocstyle: 6.3.0
- pydub: 0.25.1
- pyerfa: 2.0.1.4
- pyflakes: 3.2.0
- pygeoif: 0.7
- pygithub: 2.4.0
- pygments: 2.18.0
- pygraphviz: 1.9
- pyjwt: 2.8.0
- pylint: 3.3.1
- pylint-venv: 3.0.3
- pyls-spyder: 0.4.0
- pynacl: 1.5.0
- pynndescent: 0.5.13
- pynvml: 11.5.3
- pyodbc: 5.1.0
- pyogrio: 0.10.0
- pyparsing: 3.2.0
- pypika: 0.48.9
- pyproj: 3.6.1
- pyproject-hooks: 1.2.0
- pyqt5: 5.15.11
- pyqt5-qt5: 5.15.15
- pyqt5-sip: 12.15.0
- pyqtwebengine: 5.15.7
- pyqtwebengine-qt5: 5.15.15
- pytest: 7.4.4
- python-dateutil: 2.9.0.post0
- python-dotenv: 1.0.1
- python-hostlist: 1.23.0
- python-json-logger: 2.0.7
- python-louvain: 0.16
- python-lsp-black: 2.0.0
- python-lsp-jsonrpc: 1.1.2
- python-lsp-server: 1.12.0
- python-multipart: 0.0.12
- python-slugify: 8.0.4
- pytoolconfig: 1.2.6
- pytorch-fast-transformers: 0.4.0
- pytorch-ignite: 0.5.1
- pytorch-lightning: 2.4.0
- pytorch-msssim: 1.0.0
- pytorch-pfn-extras: 0.7.7
- pytorchvideo: 0.1.5
- pyturbojpeg: 1.7.7
- pytz: 2024.2
- pyuca: 1.2
- pyxdg: 0.28
- pyyaml: 6.0.2
- pyzmq: 26.2.0
- qdarkstyle: 3.2.3
- qdldl: 0.1.7.post4
- qstylizer: 0.2.2
- qtawesome: 1.3.1
- qtconsole: 5.6.0
- qtpy: 2.4.1
- qudida: 0.0.4
- rasterio: 1.4.1
- rawpy: 0.23.1
- ray: 2.37.0
- rdkit-pypi: 2022.9.5
- readme-renderer: 44.0
- referencing: 0.35.1
- regex: 2024.9.11
- requests: 2.32.3
- requests-oauthlib: 2.0.0
- requests-toolbelt: 1.0.0
- responses: 0.13.3
- restrictedpython: 7.4
- rfc3339-validator: 0.1.4
- rfc3986: 2.0.0
- rfc3986-validator: 0.1.1
- rich: 13.9.2
- robust-laplacian: 0.2.7
- rope: 1.13.0
- rpds-py: 0.20.0
- rsa: 4.7.2
- rtree: 1.3.0
- ruamel.yaml: 0.18.6
- ruamel.yaml.clib: 0.2.8
- ruff: 0.7.0
- runstats: 2.0.0
- s3transfer: 0.10.3
- sacremoses: 0.0.53
- safetensors: 0.4.5
- scalene: 1.5.45
- scikit-image: 0.24.0
- scikit-learn: 1.5.2
- scipy: 1.14.1
- scs: 3.2.7
- seaborn: 0.13.2
- secretstorage: 3.3.3
- segmentation-models-pytorch: 0.3.4
- semantic-version: 2.10.0
- send2trash: 1.8.3
- sentence-transformers: 3.2.0
- sentencepiece: 0.2.0
- sentinelhub: 3.11.1
- sentry-sdk: 2.17.0
- setproctitle: 1.3.3
- setuptools: 75.1.0
- shap: 0.46.0
- shapely: 2.0.6
- shellingham: 1.5.4
- shtab: 1.7.1
- simplegeneric: 0.8.1
- simpleitk: 2.4.0
- simplejson: 3.19.3
- singledispatch: 3.7.0
- six: 1.16.0
- slicer: 0.0.8
- smart-open: 7.0.5
- smmap: 5.0.1
- sniffio: 1.3.1
- snowballstemmer: 2.2.0
- softadapt: 0.0.5
- sortedcontainers: 2.4.0
- soundfile: 0.12.1
- soupsieve: 2.6
- soxr: 0.5.0.post1
- spacy: 3.7.5
- spacy-legacy: 3.0.12
- spacy-loggers: 1.0.5
- spatialite: 0.0.3
- spconv-cu120: 2.3.6
- speechbrain: 1.0.1
- sphinx: 7.3.7
- sphinxcontrib-applehelp: 2.0.0
- sphinxcontrib-devhelp: 2.0.0
- sphinxcontrib-htmlhelp: 2.1.0
- sphinxcontrib-jsmath: 1.0.1
- sphinxcontrib-qthelp: 2.0.0
- sphinxcontrib-serializinghtml: 2.0.0
- sphinxcontrib-websupport: 1.2.4
- spyder: 6.0.1
- spyder-kernels: 3.0.0
- sqlalchemy: 2.0.36
- sqlparse: 0.5.1
- srsly: 2.4.8
- stack-data: 0.6.3
- starlette: 0.40.0
- statsmodels: 0.14.4
- stringzilla: 3.10.4
- superqt: 0.6.7
- svg.path: 6.3
- swifter: 1.4.0
- sympy: 1.13.3
- tabulate: 0.9.0
- tb-nightly: 2.19.0a20241017
- tblib: 1.7.0
- tenacity: 8.5.0
- tensorboard: 2.18.0
- tensorboard-data-server: 0.7.2
- tensorboardx: 2.6.2.2
- termcolor: 2.1.0
- terminado: 0.18.1
- terminaltables: 3.1.10
- text-unidecode: 1.3
- textdistance: 4.6.3
- textual: 0.83.0
- thinc: 8.2.5
- threadpoolctl: 3.5.0
- three-merge: 0.1.1
- tifffile: 2024.9.20
- tiktoken: 0.7.0
- timm: 0.9.7
- tinycss2: 1.3.0
- tinycudann: 1.7
- tokenizers: 0.19.1
- tomli: 2.0.2
- tomli-w: 1.1.0
- tomlkit: 0.12.0
- toolwrapper: 2.1.0
- toolz: 0.12.0
- torch: 2.4.0
- torch-cluster: 1.6.3
- torch-ema: 0.3
- torch-geometric: 2.6.1
- torch-scatter: 2.1.2
- torch-sparse: 0.6.18
- torch-spline-conv: 1.2.2
- torch-tb-profiler: 0.4.3
- torchao: 0.6.1
- torchaudio: 2.4.0a0+69d4077
- torchdata: 0.8.0
- torchensemble: 0.2.0
- torchinfo: 1.8.0
- torchio: 0.20.1
- torchmetrics: 1.4.3
- torchtune: 0.3.1
- torchvision: 0.19.0a0
- torchviz: 0.0.2
- tornado: 6.4.1
- tqdm: 4.66.5
- traitlets: 5.14.3
- transformers: 4.44.1
- trimesh: 4.5.0
- triton: 3.1.0
- twine: 5.1.1
- typeguard: 4.3.0
- typer: 0.12.5
- types-dataclasses: 0.6.6
- types-python-dateutil: 2.9.0.20241003
- typing-extensions: 4.12.2
- typing-inspect: 0.9.0
- tyro: 0.8.12
- tzdata: 2024.2
- uc-micro-py: 1.0.3
- uctools: 1.3.0
- ujson: 5.10.0
- ultralytics: 8.3.15
- ultralytics-thop: 2.0.9
- umap-learn: 0.5.6
- unicodecsv: 0.14.1
- unidecode: 1.3.8
- uri-template: 1.3.0
- urllib3: 2.2.3
- utm: 0.7.0
- uvicorn: 0.32.0
- uvloop: 0.19.0
- vhacdx: 0.0.8.post1
- virtualenv: 20.27.0
- vllm-flash-attn: 2.6.1
- wandb: 0.19.7
- wasabi: 1.1.3
- watchdog: 5.0.3
- watchfiles: 0.24.0
- wcwidth: 0.2.13
- weasel: 0.4.1
- webcolors: 24.8.0
- webdataset: 0.2.100
- webencodings: 0.5.1
- websocket-client: 1.8.0
- websockets: 12.0
- werkzeug: 3.0.4
- whatthepatch: 1.0.6
- wheel: 0.44.0
- widgetsnbextension: 4.0.13
- wrapt: 1.16.0
- wurlitzer: 3.1.1
- xarray: 2024.9.0
- xatlas: 0.0.9
- xeus-python: 0.15.12
- xeus-python-shell: 0.5.0
- xformers: 0.0.27.post2
- xgboost: 2.0.0
- xlsxwriter: 3.1.1
- xlwt: 1.3.0
- xmltodict: 0.13.0
- xxhash: 2.0.2
- xyzservices: 2024.9.0
- yacs: 0.1.8
- yapf: 0.40.2
- yarl: 1.15.3
- zict: 3.0.0
- zipp: 3.20.2
* System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor: x86_64
- python: 3.11.5
- release: 5.14.0-284.55.1.el9_2.x86_64
- version: #1 SMP PREEMPT_DYNAMIC Mon Feb 19 16:57:59 EST 2024

### More info

_No response_

cc @lantiga @morganmcg1 @borisdayma @scottire @parambharat

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the WandbLogger integration and the LightningModule self.log() calls shown in the issue, reproducing the offline run and subsequent wandb sync. Trace why system logs appear while self.log() metrics do not, then add coverage demonstrating that logged loss values are available after sync.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.