Lightning-AI / Lightning-AI/pytorch-lightning
WandbLogger does not log losses with self.log() in LightningModule
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
Hello everyone,
I have been using lightning to train my model, and relying on `WandLogger` to log metrics and losses.
As a side information, I am running the code on nodes from a cluster that disables internet access for computing nodes. Thus it requires offline wandb sync after/during the training job.
I have encountered the following behavior,
- System logs are all logged on wandb and I can access them on wandb.ai without any trouble.
- However, logging the loss values on wandb does not work using `self.log()` inside my `LightningModule`. The values are correctly printed during training, but I cannot access them on wandb.ai after sync.
- When calling `self.logger.experiment.log()`, the loss values are correctly logged and accessible on wandb.ai.
### What version are you seeing the problem on?
v2.4
### How to reproduce the bug
The main file:
```python
...
wandb_logger = WandbLogger(
project="ProjectName",
offline=True,
config=config,)
...
trainer = pl.Trainer(
max_epochs=config['training']['num_epochs'],
accumulate_grad_batches=config['training']['gradient_accumulation_steps'],
callbacks=callbacks,
devices=int(os.environ['SLURM_GPUS_ON_NODE']),
accelerator=device_type,
strategy=strategy,
logger=wandb_logger,
profiler=profiler,
enable_model_summary=True,
num_nodes=int(os.environ['SLURM_NNODES']),
detect_anomaly=config['anomaly_run'],
fast_dev_run=config['fast_dev_run'],
)
trainer.fit(pl_model,
dataset,
ckpt_path=load_pth)
```
The`LightningModule`:
```python
import lightning as pl
class MyClass(pl.LightningModule):
def __init__(self,
model: nn.Module,
config: Dict):
...
def training_step(self, batch, batch_idx):
...
asr_loss = ...
em_loss = ...
total_loss = asr_loss + em_loss
self.log("Train/ASR_step_loss",
asr_loss.item(),
prog_bar=True,
on_step=True,
on_epoch=True,
sync_dist=True,
logger=True,)
self.log("Train/total_loss",
total_loss.item(),
prog_bar=True,
on_step=True,
on_epoch=True,
sync_dist=True,
logger=True,)
self.log("Train/EM_step_loss",
loss_em.item(),
prog_bar=True,
on_step=True,
on_epoch=True,
sync_dist=True,
logger=True,)
# without this code, loss are not logged on wandb
self.logger.experiment.log(
{"Train/ASR_step_loss":
asr_loss.item()},
)
self.logger.experiment.log(
{"Train/em_loss":
em_loss.item()},
)
self.logger.experiment.log(
{"Train/total_loss":
total_loss.item()},
)
return total_loss
```
### Error messages and logs
As a side note, the wandb logs provide the following:
```
2025-02-21 11:02:09,547 ERROR MainThread:3724374 [wandb_run.py:_redirect():2320] Failed to redirect.
Traceback (most recent call last):
File "path/to/.local/lib/python3.11/site-packages/wandb/sdk/wandb_run.py", line 2313, in _redirect
out_redir.install()
File "path/to/.local/lib/python3.11/site-packages/wandb/sdk/lib/redirect.py", line 654, in install
stream.write = write
^^^^^^^^^^^^
AttributeError: 'stream.Stream' object attribute 'write' is read-only
2025-02-21 11:02:09,556 INFO MainThread:3724374 [wandb_init.py:init():1037] run started, returning control to user process
```
### Environment
Current environment
* CUDA:
- GPU:
- Tesla V100-SXM2-16GB
- available: True
- version: 12.2
* Lightning:
- efficientnet-pytorch: 0.7.1
- idr-torch: 2.2.0
- lightning: 2.4.0
- lightning-utilities: 0.11.8
- pytorch-fast-transformers: 0.4.0
- pytorch-ignite: 0.5.1
- pytorch-lightning: 2.4.0
- pytorch-msssim: 1.0.0
- pytorch-pfn-extras: 0.7.7
- pytorchvideo: 0.1.5
- segmentation-models-pytorch: 0.3.4
- torch: 2.4.0
- torch-cluster: 1.6.3
- torch-ema: 0.3
- torch-geometric: 2.6.1
- torch-scatter: 2.1.2
- torch-sparse: 0.6.18
- torch-spline-conv: 1.2.2
- torch-tb-profiler: 0.4.3
- torchao: 0.6.1
- torchaudio: 2.4.0a0+69d4077
- torchdata: 0.8.0
- torchensemble: 0.2.0
- torchinfo: 1.8.0
- torchio: 0.20.1
- torchmetrics: 1.4.3
- torchtune: 0.3.1
- torchvision: 0.19.0a0
- torchviz: 0.0.2
* Packages:
- absl-py: 2.1.0
- accelerate: 1.0.1
- addict: 2.4.0
- aenum: 3.1.15
- affine: 2.4.0
- aim: 3.25.0
- aim-ui: 3.25.0
- aimrecords: 0.0.7
- aimrocks: 0.5.2
- aiofiles: 23.2.1
- aiohappyeyeballs: 2.4.3
- aiohttp: 3.10.10
- aiohttp-cors: 0.7.0
- aiosignal: 1.3.1
- alabaster: 0.7.16
- albucore: 0.0.18
- albumentations: 1.4.18
- alembic: 1.13.3
- aniso8601: 9.0.1
- annotated-types: 0.7.0
- antlr4-python3-runtime: 4.9.3
- anyio: 4.2.0
- apex: 0.1
- appdirs: 1.4.4
- argon2-cffi: 23.1.0
- argon2-cffi-bindings: 21.2.0
- arrow: 1.3.0
- asgiref: 3.8.1
- asn1crypto: 1.5.1
- assertpy: 1.1
- astroid: 3.3.5
- astropy: 6.1.4
- astropy-iers-data: 0.2024.10.14.0.32.55
- asttokens: 2.4.1
- astunparse: 1.6.3
- async-lru: 2.0.4
- asyncssh: 2.17.0
- atomicwrites: 1.4.1
- attrs: 24.2.0
- audioread: 3.0.1
- autocommand: 2.2.2
- autopep8: 2.0.4
- av: 13.1.0
- awscli: 1.35.8
- babel: 2.16.0
- backoff: 2.2.1
- backports.tarfile: 1.2.0
- base58: 2.0.1
- basicsr: 1.4.2
- bcrypt: 4.2.0
- beautifulsoup4: 4.12.3
- binaryornot: 0.4.4
- biopython: 1.84
- bitarray: 2.5.1
- bitsandbytes: 0.44.1
- bkcharts: 0.2
- black: 24.10.0
- bleach: 6.1.0
- blinker: 1.6.2
- blis: 0.7.11
- blobfile: 3.0.0
- bokeh: 3.6.0
- boto: 2.49.0
- boto3: 1.35.42
- botocore: 1.35.42
- bottleneck: 1.4.1
- braceexpand: 0.1.7
- branca: 0.8.0
- build: 1.2.2.post1
- cachetools: 5.5.0
- catalogue: 2.0.10
- ccimport: 0.4.4
- certifi: 2024.8.30
- cffi: 1.17.1
- chardet: 5.2.0
- charset-normalizer: 3.4.0
- chroma-hnswlib: 0.7.6
- chromadb: 0.5.15
- clarabel: 0.9.0
- click: 8.1.7
- click-plugins: 1.1.1
- cligj: 0.7.2
- clip: 0.2.0
- cloudpathlib: 0.19.0
- cloudpickle: 3.1.0
- cmake: 3.30.4
- colorama: 0.4.6
- coloredlogs: 15.0.1
- colorful: 0.5.6
- colorlog: 6.8.2
- comm: 0.2.2
- compressai: 1.2.6
- confection: 0.1.5
- contourpy: 1.3.0
- cookiecutter: 2.6.0
- cryptography: 43.0.1
- cumm-cu120: 0.4.11
- cupy-cuda12x: 13.3.0
- cvxpy: 1.5.3
- cvxpylayers: 0.1.6
- cxxfilt: 0.3.0
- cycler: 0.12.1
- cymem: 2.0.8
- cytoolz: 0.12.2
- dask: 2024.8.2
- dask-expr: 1.1.13
- dask-jobqueue: 0.8.2
- databricks-sdk: 0.35.0
- dataclasses: 0.6
- dataclasses-json: 0.6.7
- datasets: 2.12.0
- debugpy: 1.8.7
- decorator: 5.1.1
- deepspeed: 0.15.2
- defusedxml: 0.7.1
- deprecated: 1.2.13
- dgl: 2.1.0
- dglgo: 0.0.2
- diff-match-patch: 20230430
- diffcp: 1.0.23
- diffusers: 0.30.3
- dill: 0.3.6
- diskcache: 5.6.3
- distarray: 0.6.0
- distlib: 0.3.9
- distributed: 2024.8.2
- distro: 1.9.0
- dm-tree: 0.1.8
- docker: 7.1.0
- docker-pycreds: 0.4.0
- docopt: 0.6.2
- docstring-parser: 0.16
- docstring-to-markdown: 0.15
- docutils: 0.21.2
- durationpy: 0.9
- eagerpy: 0.30.0
- ecos: 2.0.14
- efficientnet-pytorch: 0.7.1
- einops: 0.8.0
- embreex: 2.17.7.post5
- entrypoints: 0.4
- eo-learn: 1.5.7
- eo-learn-core: 1.5.0
- eo-learn-coregistration: 1.5.0
- eo-learn-features: 1.5.0
- eo-learn-geometry: 1.5.0
- eo-learn-io: 1.5.0
- eo-learn-mask: 1.5.0
- eo-learn-ml-tools: 1.5.0
- eo-learn-visualization: 1.5.0
- evaluate: 0.4.3
- executing: 2.1.0
- expecttest: 0.2.1
- fabric: 2.6.0
- fastai: 2.7.17
- fastapi: 0.115.2
- fastargs: 1.2.0
- fastcore: 1.7.19
- fastdownload: 0.0.7
- fastjsonschema: 2.20.0
- fastkml: 0.12
- fastmri: 0.3.0
- fastprogress: 1.0.3
- fastrlock: 0.8.2
- ffcv: 1.0.2
- ffmpy: 0.4.0
- filelock: 3.13.1
- filetype: 1.2.0
- findiff: 0.10.2
- fiona: 1.10.1
- fire: 0.7.0
- flake8: 7.1.1
- flash-attn: 2.6.1
- flask: 3.0.3
- flatbuffers: 24.3.25
- folium: 0.17.0
- fonttools: 4.54.1
- foolbox: 3.3.4
- fqdn: 1.5.1
- frozenlist: 1.4.1
- fs: 2.4.16
- fs-s3fs: 1.1.1
- fsspec: 2023.6.0
- future: 1.0.0
- fvcore: 0.1.6
- gast: 0.5.3
- geographiclib: 2.0
- geopandas: 0.14.4
- gguf: 0.10.0
- gitdb: 4.0.11
- gitpython: 3.1.43
- glob2: 0.7
- global-land-mask: 1.0.0
- gmpy2: 2.1.2
- google-api-core: 2.21.0
- google-auth: 2.35.0
- googleapis-common-protos: 1.65.0
- gradio: 5.1.0
- gradio-client: 1.4.0
- graphene: 3.3
- graphql-core: 3.2.5
- graphql-relay: 3.2.0
- graphviz: 0.20.3
- greenlet: 3.1.1
- grpcio: 1.67.0
- gunicorn: 23.0.0
- guppy3: 3.1.4.post1
- h11: 0.14.0
- h5py: 3.12.1
- heapdict: 1.0.1
- hjson: 3.1.0
- horovod: 0.28.1
- httpcore: 1.0.6
- httptools: 0.6.4
- httpx: 0.27.2
- huggingface-hub: 0.26.0
- humanfriendly: 10.0
- humanize: 4.11.0
- hydra-core: 1.3.2
- hyperpyyaml: 1.2.2
- hypothesis: 6.115.2
- idna: 3.7
- idr-accelerate: 2.0.1
- idr-env-protect: 2.0.0
- idr-pypi: 1.0.0
- idr-torch: 2.2.0
- idris: 1.0.0
- imageio: 2.33.1
- imageio-ffmpeg: 0.5.1
- imagesize: 1.4.1
- imbalanced-learn: 0.12.4
- importlib-metadata: 8.4.0
- importlib-resources: 6.4.0
- imutils: 0.5.4
- inflect: 7.3.1
- inflection: 0.5.1
- iniconfig: 1.1.1
- interegular: 0.3.3
- intervaltree: 3.1.0
- invoke: 1.6.0
- iopath: 0.1.10
- ipykernel: 6.29.5
- ipyparallel: 8.8.0
- ipython: 8.28.0
- ipython-genutils: 0.2.0
- ipywidgets: 8.1.5
- isoduration: 20.11.0
- isort: 5.13.2
- itsdangerous: 2.2.0
- jaraco.classes: 3.4.0
- jaraco.collections: 5.1.0
- jaraco.context: 6.0.1
- jaraco.functools: 4.1.0
- jaraco.text: 3.12.1
- jedi: 0.19.1
- jeepney: 0.8.0
- jellyfish: 1.1.0
- jinja2: 3.1.4
- jiter: 0.6.1
- jmespath: 1.0.1
- joblib: 1.4.2
- json5: 0.9.25
- jsonpatch: 1.33
- jsonpointer: 3.0.0
- jsonschema: 4.23.0
- jsonschema-specifications: 2024.10.1
- jupyter: 1.1.1
- jupyter-client: 8.6.0
- jupyter-console: 6.6.3
- jupyter-contrib-core: 0.4.2
- jupyter-core: 5.7.2
- jupyter-events: 0.10.0
- jupyter-lsp: 2.2.5
- jupyter-nbextensions-configurator: 0.6.4
- jupyter-server: 2.14.1
- jupyter-server-terminals: 0.4.4
- jupyter-tensorboard: 0.2.0
- jupyterlab: 4.2.5
- jupyterlab-pygments: 0.1.2
- jupyterlab-server: 2.27.3
- jupyterlab-widgets: 3.0.13
- jupyterquiz: 2.6.3
- keyring: 25.4.1
- kiwisolver: 1.4.7
- kornia: 0.7.3
- kornia-rs: 0.1.5
- kubernetes: 31.0.0
- langchain: 0.3.3
- langchain-core: 0.3.12
- langchain-text-splitters: 0.3.0
- langcodes: 3.4.1
- langsmith: 0.1.136
- language-data: 1.2.0
- lapjv: 1.3.27
- lark: 1.2.2
- lazy-loader: 0.4
- librosa: 0.10.2.post1
- lief: 0.15.1
- lightgbm: 4.5.0
- lightning: 2.4.0
- lightning-utilities: 0.11.8
- linkify-it-py: 2.0.3
- lintrunner: 0.12.5
- littleutils: 0.2.4
- llvmlite: 0.43.0
- lm-format-enforcer: 0.10.6
- lmdb: 1.4.1
- locket: 1.0.0
- lpips: 0.1.4
- lxml: 5.2.1
- mako: 1.3.5
- manifold3d: 2.5.1
- mapbox-earcut: 1.0.2
- marisa-trie: 1.2.1
- markdown: 3.7
- markdown-it-py: 3.0.0
- markupsafe: 2.1.5
- marshmallow: 3.22.0
- matplotlib: 3.9.2
- matplotlib-inline: 0.1.7
- mccabe: 0.7.0
- mdit-py-plugins: 0.4.2
- mdurl: 0.1.2
- megatron-lm: 3.0.0
- memray: 1.14.0
- mistral-common: 1.4.4
- mistune: 3.0.2
- mizani: 0.11.4
- mlconfig: 0.2.2
- mlflow: 2.17.0
- mlflow-skinny: 2.17.0
- mmcv: 2.2.0
- mmengine: 0.10.5
- mmh3: 5.0.1
- mock: 4.0.3
- monai: 1.3.1
- monotonic: 1.6
- more-itertools: 10.5.0
- moses: 0.10.0
- mosestokenizer: 1.2.1
- mpi4py: 4.0.1
- mpmath: 1.3.0
- msgpack: 1.0.3
- msgspec: 0.18.6
- multidict: 6.1.0
- multipledispatch: 0.6.0
- multiprocess: 0.70.14
- munch: 4.0.0
- murmurhash: 1.0.10
- mypy-extensions: 1.0.0
- natsort: 8.4.0
- nbclient: 0.10.0
- nbconvert: 7.16.4
- nbformat: 5.10.4
- nest-asyncio: 1.6.0
- networkx: 3.2.1
- nh3: 0.2.18
- nibabel: 5.3.1
- ninja: 1.11.1.1
- nltk: 3.9.1
- notebook: 6.4.12
- notebook-shim: 0.2.4
- numba: 0.60.0
- numpy: 1.26.4
- numpydoc: 1.8.0
- nvidia-dali-cuda120: 1.42.0
- nvidia-dlprof: 0.17.0
- nvidia-ml-py: 12.560.30
- nvidia-nsys-cli: 2021.3.2.12
- nvidia-nvimgcodec-cu12: 0.3.0.5
- nvidia-pyprof: 3.10.0
- oauthlib: 3.2.2
- ogb: 1.3.6
- olefile: 0.46
- omegaconf: 2.3.0
- onnxruntime: 1.19.2
- openai: 1.51.2
- openai-whisper: 20240930
- opencensus: 0.11.4
- opencensus-context: 0.1.3
- opencv-python: 4.10.0.84
- opencv-python-headless: 4.10.0.84
- openfile: 0.0.7
- opentelemetry-api: 1.27.0
- opentelemetry-exporter-otlp-proto-common: 1.27.0
- opentelemetry-exporter-otlp-proto-grpc: 1.27.0
- opentelemetry-instrumentation: 0.48b0
- opentelemetry-instrumentation-asgi: 0.48b0
- opentelemetry-instrumentation-fastapi: 0.48b0
- opentelemetry-proto: 1.27.0
- opentelemetry-sdk: 1.27.0
- opentelemetry-semantic-conventions: 0.48b0
- opentelemetry-util-http: 0.48b0
- opt-einsum: 3.4.0
- optree: 0.13.0
- optuna: 4.0.0
- orjson: 3.10.7
- osqp: 0.6.7.post3
- outdated: 0.2.2
- outlines: 0.0.46
- overrides: 7.7.0
- packaging: 24.1
- pafit: 2.0.8
- pandas: 2.2.3
- pandocfilters: 1.5.1
- parameterized: 0.9.0
- paramiko: 2.8.1
- parso: 0.8.4
- partd: 1.4.1
- partial-json-parser: 0.2.1.1.post4
- pathlib2: 2.3.6
- pathspec: 0.12.1
- patsy: 0.5.6
- pccm: 0.4.16
- peft: 0.13.2
- pexpect: 4.9.0
- photutils: 2.0.1
- pickleshare: 0.7.5
- pillow: 10.4.0
- pip: 24.2
- pkginfo: 1.10.0
- platformdirs: 3.10.0
- plotbin: 3.1.7
- plotly: 5.24.1
- plotnine: 0.13.6
- pluggy: 1.5.0
- ply: 3.11
- plyfile: 1.1
- pooch: 1.8.2
- portalocker: 2.10.1
- posthog: 3.7.0
- potpourri3d: 1.1.0
- preshed: 3.0.9
- pretrainedmodels: 0.7.4
- prettytable: 3.14.0
- prometheus-client: 0.14.1
- prometheus-fastapi-instrumentator: 7.0.0
- prompt-toolkit: 3.0.48
- propcache: 0.2.0
- proto-plus: 1.24.0
- protobuf: 4.25.5
- psutil: 6.0.0
- ptyprocess: 0.7.0
- pure-eval: 0.2.3
- py-cpuinfo: 9.0.0
- py-spy: 0.3.14
- pyairports: 2.1.1
- pyarrow: 17.0.0
- pyasn1: 0.6.1
- pyasn1-modules: 0.4.1
- pybind11: 2.12.0
- pybind11-global: 2.12.0
- pycairo: 1.23.0
- pycodestyle: 2.12.1
- pycollada: 0.8
- pyconll: 3.2.0
- pycountry: 24.6.1
- pycparser: 2.22
- pycryptodomex: 3.21.0
- pycurl: 7.45.3
- pydantic: 2.9.2
- pydantic-core: 2.23.4
- pydocstyle: 6.3.0
- pydub: 0.25.1
- pyerfa: 2.0.1.4
- pyflakes: 3.2.0
- pygeoif: 0.7
- pygithub: 2.4.0
- pygments: 2.18.0
- pygraphviz: 1.9
- pyjwt: 2.8.0
- pylint: 3.3.1
- pylint-venv: 3.0.3
- pyls-spyder: 0.4.0
- pynacl: 1.5.0
- pynndescent: 0.5.13
- pynvml: 11.5.3
- pyodbc: 5.1.0
- pyogrio: 0.10.0
- pyparsing: 3.2.0
- pypika: 0.48.9
- pyproj: 3.6.1
- pyproject-hooks: 1.2.0
- pyqt5: 5.15.11
- pyqt5-qt5: 5.15.15
- pyqt5-sip: 12.15.0
- pyqtwebengine: 5.15.7
- pyqtwebengine-qt5: 5.15.15
- pytest: 7.4.4
- python-dateutil: 2.9.0.post0
- python-dotenv: 1.0.1
- python-hostlist: 1.23.0
- python-json-logger: 2.0.7
- python-louvain: 0.16
- python-lsp-black: 2.0.0
- python-lsp-jsonrpc: 1.1.2
- python-lsp-server: 1.12.0
- python-multipart: 0.0.12
- python-slugify: 8.0.4
- pytoolconfig: 1.2.6
- pytorch-fast-transformers: 0.4.0
- pytorch-ignite: 0.5.1
- pytorch-lightning: 2.4.0
- pytorch-msssim: 1.0.0
- pytorch-pfn-extras: 0.7.7
- pytorchvideo: 0.1.5
- pyturbojpeg: 1.7.7
- pytz: 2024.2
- pyuca: 1.2
- pyxdg: 0.28
- pyyaml: 6.0.2
- pyzmq: 26.2.0
- qdarkstyle: 3.2.3
- qdldl: 0.1.7.post4
- qstylizer: 0.2.2
- qtawesome: 1.3.1
- qtconsole: 5.6.0
- qtpy: 2.4.1
- qudida: 0.0.4
- rasterio: 1.4.1
- rawpy: 0.23.1
- ray: 2.37.0
- rdkit-pypi: 2022.9.5
- readme-renderer: 44.0
- referencing: 0.35.1
- regex: 2024.9.11
- requests: 2.32.3
- requests-oauthlib: 2.0.0
- requests-toolbelt: 1.0.0
- responses: 0.13.3
- restrictedpython: 7.4
- rfc3339-validator: 0.1.4
- rfc3986: 2.0.0
- rfc3986-validator: 0.1.1
- rich: 13.9.2
- robust-laplacian: 0.2.7
- rope: 1.13.0
- rpds-py: 0.20.0
- rsa: 4.7.2
- rtree: 1.3.0
- ruamel.yaml: 0.18.6
- ruamel.yaml.clib: 0.2.8
- ruff: 0.7.0
- runstats: 2.0.0
- s3transfer: 0.10.3
- sacremoses: 0.0.53
- safetensors: 0.4.5
- scalene: 1.5.45
- scikit-image: 0.24.0
- scikit-learn: 1.5.2
- scipy: 1.14.1
- scs: 3.2.7
- seaborn: 0.13.2
- secretstorage: 3.3.3
- segmentation-models-pytorch: 0.3.4
- semantic-version: 2.10.0
- send2trash: 1.8.3
- sentence-transformers: 3.2.0
- sentencepiece: 0.2.0
- sentinelhub: 3.11.1
- sentry-sdk: 2.17.0
- setproctitle: 1.3.3
- setuptools: 75.1.0
- shap: 0.46.0
- shapely: 2.0.6
- shellingham: 1.5.4
- shtab: 1.7.1
- simplegeneric: 0.8.1
- simpleitk: 2.4.0
- simplejson: 3.19.3
- singledispatch: 3.7.0
- six: 1.16.0
- slicer: 0.0.8
- smart-open: 7.0.5
- smmap: 5.0.1
- sniffio: 1.3.1
- snowballstemmer: 2.2.0
- softadapt: 0.0.5
- sortedcontainers: 2.4.0
- soundfile: 0.12.1
- soupsieve: 2.6
- soxr: 0.5.0.post1
- spacy: 3.7.5
- spacy-legacy: 3.0.12
- spacy-loggers: 1.0.5
- spatialite: 0.0.3
- spconv-cu120: 2.3.6
- speechbrain: 1.0.1
- sphinx: 7.3.7
- sphinxcontrib-applehelp: 2.0.0
- sphinxcontrib-devhelp: 2.0.0
- sphinxcontrib-htmlhelp: 2.1.0
- sphinxcontrib-jsmath: 1.0.1
- sphinxcontrib-qthelp: 2.0.0
- sphinxcontrib-serializinghtml: 2.0.0
- sphinxcontrib-websupport: 1.2.4
- spyder: 6.0.1
- spyder-kernels: 3.0.0
- sqlalchemy: 2.0.36
- sqlparse: 0.5.1
- srsly: 2.4.8
- stack-data: 0.6.3
- starlette: 0.40.0
- statsmodels: 0.14.4
- stringzilla: 3.10.4
- superqt: 0.6.7
- svg.path: 6.3
- swifter: 1.4.0
- sympy: 1.13.3
- tabulate: 0.9.0
- tb-nightly: 2.19.0a20241017
- tblib: 1.7.0
- tenacity: 8.5.0
- tensorboard: 2.18.0
- tensorboard-data-server: 0.7.2
- tensorboardx: 2.6.2.2
- termcolor: 2.1.0
- terminado: 0.18.1
- terminaltables: 3.1.10
- text-unidecode: 1.3
- textdistance: 4.6.3
- textual: 0.83.0
- thinc: 8.2.5
- threadpoolctl: 3.5.0
- three-merge: 0.1.1
- tifffile: 2024.9.20
- tiktoken: 0.7.0
- timm: 0.9.7
- tinycss2: 1.3.0
- tinycudann: 1.7
- tokenizers: 0.19.1
- tomli: 2.0.2
- tomli-w: 1.1.0
- tomlkit: 0.12.0
- toolwrapper: 2.1.0
- toolz: 0.12.0
- torch: 2.4.0
- torch-cluster: 1.6.3
- torch-ema: 0.3
- torch-geometric: 2.6.1
- torch-scatter: 2.1.2
- torch-sparse: 0.6.18
- torch-spline-conv: 1.2.2
- torch-tb-profiler: 0.4.3
- torchao: 0.6.1
- torchaudio: 2.4.0a0+69d4077
- torchdata: 0.8.0
- torchensemble: 0.2.0
- torchinfo: 1.8.0
- torchio: 0.20.1
- torchmetrics: 1.4.3
- torchtune: 0.3.1
- torchvision: 0.19.0a0
- torchviz: 0.0.2
- tornado: 6.4.1
- tqdm: 4.66.5
- traitlets: 5.14.3
- transformers: 4.44.1
- trimesh: 4.5.0
- triton: 3.1.0
- twine: 5.1.1
- typeguard: 4.3.0
- typer: 0.12.5
- types-dataclasses: 0.6.6
- types-python-dateutil: 2.9.0.20241003
- typing-extensions: 4.12.2
- typing-inspect: 0.9.0
- tyro: 0.8.12
- tzdata: 2024.2
- uc-micro-py: 1.0.3
- uctools: 1.3.0
- ujson: 5.10.0
- ultralytics: 8.3.15
- ultralytics-thop: 2.0.9
- umap-learn: 0.5.6
- unicodecsv: 0.14.1
- unidecode: 1.3.8
- uri-template: 1.3.0
- urllib3: 2.2.3
- utm: 0.7.0
- uvicorn: 0.32.0
- uvloop: 0.19.0
- vhacdx: 0.0.8.post1
- virtualenv: 20.27.0
- vllm-flash-attn: 2.6.1
- wandb: 0.19.7
- wasabi: 1.1.3
- watchdog: 5.0.3
- watchfiles: 0.24.0
- wcwidth: 0.2.13
- weasel: 0.4.1
- webcolors: 24.8.0
- webdataset: 0.2.100
- webencodings: 0.5.1
- websocket-client: 1.8.0
- websockets: 12.0
- werkzeug: 3.0.4
- whatthepatch: 1.0.6
- wheel: 0.44.0
- widgetsnbextension: 4.0.13
- wrapt: 1.16.0
- wurlitzer: 3.1.1
- xarray: 2024.9.0
- xatlas: 0.0.9
- xeus-python: 0.15.12
- xeus-python-shell: 0.5.0
- xformers: 0.0.27.post2
- xgboost: 2.0.0
- xlsxwriter: 3.1.1
- xlwt: 1.3.0
- xmltodict: 0.13.0
- xxhash: 2.0.2
- xyzservices: 2024.9.0
- yacs: 0.1.8
- yapf: 0.40.2
- yarl: 1.15.3
- zict: 3.0.0
- zipp: 3.20.2
* System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor: x86_64
- python: 3.11.5
- release: 5.14.0-284.55.1.el9_2.x86_64
- version: #1 SMP PREEMPT_DYNAMIC Mon Feb 19 16:57:59 EST 2024
### More info
_No response_
cc @lantiga @morganmcg1 @borisdayma @scottire @parambharat
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the WandbLogger integration and the LightningModule self.log() calls shown in the issue, reproducing the offline run and subsequent wandb sync. Trace why system logs appear while self.log() metrics do not, then add coverage demonstrating that logged loss values are available after sync.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100