Lightning-AI / Lightning-AI/pytorch-lightning

LightningCLI: incorrect default value of kwarg used

Open
#18,616 13 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

3rd party bug lightningcli ver: 2.0.x
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

I'm trying to port TorchGeo to LightningCLI: https://github.com/microsoft/torchgeo/pull/1559

99% of things are great! Unfortunately I'm still stuck on the last 1%. If you look at the [failing tests](https://github.com/microsoft/torchgeo/actions/runs/6277244109/job/17048954111), you can see two issues:

### So2SatDataModule

When parsing [So2SatDataModule](https://github.com/microsoft/torchgeo/blob/main/torchgeo/datamodules/so2sat.py#L16), jsonargparse notices that kwargs may contain a `version` parameter. If it doesn't, the default is "2". However, jsonargparse detects the type as int and uses a default value of 2. Since "2" is in the dict but 2 isn't, the dataset fails to initialize.

### SeasonalContrastS2DataModule

When parsing [SeasonalContrastS2DataModule](https://github.com/microsoft/torchgeo/blob/main/torchgeo/datamodules/seco.py#L17), jsonargparse notices that kwargs may contain a `bands` parameter. If it doesn't, the default is `["B4", "B3", "B2"]`. However, jsonargparse detects the type as `Any` and uses a default value of None. Obviously, None is invalid.

It's still not clear to me if this is a lightning issue or a jsonargparse issue, but I'm surprised that it's trying to poke into my kwargs at all. Isn't that the point of `dict_kwargs`, for things that can't be type checked? Even more surprising is that even though my [seasonal contrast](https://github.com/microsoft/torchgeo/blob/main/tests/conf/seco_byol_1.yaml) and [so2sat](https://github.com/microsoft/torchgeo/blob/main/tests/conf/so2sat_s1.yaml) config.yaml files don't use these parameters, lightning is adding a default value anyway.

Is this a bug in jsonarpgarse or lightning? Is there any way to disable this aggressive kwarg parsing or use of default values?

### What version are you seeing the problem on?

v2.0

### How to reproduce the bug

_No response_

### Error messages and logs

_No response_

### Environment

Current environment

* CUDA:
- GPU: None
- available: False
- version: None
* Lightning:
- efficientnet-pytorch: 0.7.1
- lightning: 2.0.9
- lightning-cloud: 0.5.38
- lightning-utilities: 0.8.0
- pytorch-lightning: 2.0.0
- pytorch-sphinx-theme: 0.0.24
- segmentation-models-pytorch: 0.3.3
- torch: 2.0.1
- torchmetrics: 1.1.1
- torchvision: 0.15.2
* Packages:
- absl-py: 1.4.0
- aenum: 3.1.12
- affine: 2.1.0
- aiohttp: 3.8.4
- aiosignal: 1.2.0
- alabaster: 0.7.13
- antlr4-python3-runtime: 4.9.3
- anyio: 3.6.2
- appdirs: 1.4.4
- appnope: 0.1.3
- argon2-cffi: 21.3.0
- argon2-cffi-bindings: 21.2.0
- arrow: 1.2.3
- asttokens: 2.2.1
- astunparse: 1.6.3
- async-lru: 1.0.3
- async-timeout: 4.0.2
- attrs: 23.1.0
- babel: 2.12.1
- backcall: 0.2.0
- backoff: 2.2.1
- beautifulsoup4: 4.12.2
- black: 23.9.1
- bleach: 6.0.0
- blessed: 1.19.0
- bottleneck: 1.3.7
- build: 1.0.3
- cachetools: 5.2.0
- cartopy: 0.22.0
- certifi: 2023.5.7
- cffi: 1.15.1
- cftime: 1.0.3.4
- charset-normalizer: 3.1.0
- click: 8.1.3
- click-plugins: 1.1.1
- cligj: 0.7.2
- cmocean: 2.0
- colorama: 0.4.6
- comm: 0.1.3
- contourpy: 1.0.7
- coverage: 7.2.6
- croniter: 1.3.8
- cycler: 0.11.0
- cython: 0.29.36
- dateutils: 0.6.12
- debugpy: 1.6.7
- decorator: 5.1.1
- deepdiff: 6.3.0
- defusedxml: 0.7.1
- docstring-parser: 0.15
- docutils: 0.18.1
- editables: 0.3
- efficientnet-pytorch: 0.7.1
- einops: 0.6.1
- et-xmlfile: 1.0.1
- executing: 1.2.0
- fastapi: 0.98.0
- fastjsonschema: 2.16.3
- filelock: 3.12.0
- fiona: 1.9.4
- flake8: 6.1.0
- flit-core: 3.9.0
- fonttools: 4.39.4
- fqdn: 1.5.1
- frozenlist: 1.3.1
- fsspec: 2023.1.0
- gdal: 3.7.2
- geocube: 0.3.2
- geopandas: 0.11.1
- gevent: 23.7.0
- google-auth: 2.20.0
- google-auth-oauthlib: 0.5.2
- greenlet: 2.0.2
- grpcio: 1.52.0
- h11: 0.13.0
- h5py: 3.8.0
- hatch-jupyter-builder: 0.8.3
- hatchling: 1.17.0
- huggingface-hub: 0.14.1
- hydra-core: 1.3.1
- idna: 3.4
- imagesize: 1.4.1
- importlib-metadata: 6.6.0
- importlib-resources: 5.12.0
- iniconfig: 2.0.0
- inquirer: 3.1.3
- ipykernel: 6.23.1
- ipython: 8.14.0
- ipywidgets: 8.0.2
- isoduration: 20.11.0
- isort: 5.12.0
- itsdangerous: 2.1.2
- jaraco.classes: 3.2.3
- jedi: 0.18.2
- jinja2: 3.0.3
- joblib: 1.2.0
- json5: 0.9.14
- jsonargparse: 4.19.0
- jsonpointer: 2.0
- jsonschema: 4.17.3
- jupyter-client: 8.2.0
- jupyter-core: 5.3.0
- jupyter-events: 0.6.3
- jupyter-lsp: 2.2.0
- jupyter-server: 2.6.0
- jupyter-server-terminals: 0.4.4
- jupyterlab: 4.0.1
- jupyterlab-pygments: 0.2.2
- jupyterlab-server: 2.22.1
- jupyterlab-widgets: 3.0.3
- keyring: 23.13.1
- kiwisolver: 1.4.4
- kornia: 0.7.0
- laspy: 2.2.0
- lightly: 1.4.18
- lightly-utils: 0.0.2
- lightning: 2.0.9
- lightning-cloud: 0.5.38
- lightning-utilities: 0.8.0
- markdown: 3.4.1
- markdown-it-py: 3.0.0
- markupsafe: 2.1.3
- matplotlib: 3.8.0
- matplotlib-inline: 0.1.6
- mccabe: 0.7.0
- mdurl: 0.1.2
- mistune: 2.0.5
- more-itertools: 9.1.0
- mpmath: 1.2.1
- multidict: 6.0.4
- munch: 2.5.0
- mypy: 1.3.0
- mypy-extensions: 1.0.0
- nbclient: 0.6.7
- nbconvert: 7.4.0
- nbformat: 5.8.0
- nbmake: 1.4.3
- nbsphinx: 0.8.8
- nest-asyncio: 1.5.6
- netcdf4: 1.6.2
- networkx: 3.1
- notebook-shim: 0.2.3
- numexpr: 2.8.4
- numpy: 1.25.2
- oauthlib: 3.2.1
- odc-geo: 0.1.2
- omegaconf: 2.3.0
- openpyxl: 3.1.2
- ordered-set: 4.0.2
- overrides: 7.3.1
- packaging: 23.1
- pandas: 2.0.2
- pandocfilters: 1.5.0
- parso: 0.8.3
- pathspec: 0.11.1
- pexpect: 4.8.0
- pickleshare: 0.7.5
- pillow: 9.5.0
- pip: 23.0
- pkginfo: 1.9.6
- planetary-computer: 0.4.9
- platformdirs: 3.5.3
- pluggy: 1.0.0
- poetry-core: 1.6.1
- pretrainedmodels: 0.7.4
- prometheus-client: 0.17.0
- prompt-toolkit: 3.0.38
- protobuf: 3.20.3
- psutil: 5.9.5
- ptyprocess: 0.7.0
- pure-eval: 0.2.2
- pyasn1: 0.4.8
- pyasn1-modules: 0.2.8
- pybind11: 2.10.1
- pycocotools: 2.0.6
- pycodestyle: 2.11.0
- pycparser: 2.21
- pydantic: 1.10.9
- pydocstyle: 6.2.1
- pyflakes: 3.1.0
- pygeos: 0.10
- pygments: 2.15.1
- pyjwt: 2.4.0
- pyparsing: 3.0.9
- pyproj: 3.6.0
- pyproject-hooks: 1.0.0
- pyrsistent: 0.19.3
- pyshp: 2.1.0
- pystac: 1.4.0
- pystac-client: 0.5.1
- pytest: 7.3.2
- pytest-cov: 4.0.0
- python-dateutil: 2.8.2
- python-dotenv: 0.19.2
- python-editor: 1.0.4
- python-json-logger: 2.0.7
- python-multipart: 0.0.5
- pytorch-lightning: 2.0.0
- pytorch-sphinx-theme: 0.0.24
- pytz: 2023.3
- pyupgrade: 3.3.1
- pyyaml: 6.0
- pyzmq: 25.0.2
- radiant-mlhub: 0.5.1
- rarfile: 4.1
- rasterio: 1.3.8
- readchar: 4.0.5
- readme-renderer: 37.3
- requests: 2.31.0
- requests-oauthlib: 1.3.1
- requests-toolbelt: 1.0.0
- rfc3339-validator: 0.1.4
- rfc3986: 2.0.0
- rfc3986-validator: 0.1.1
- rich: 13.4.2
- rioxarray: 0.4.1.post0
- rsa: 4.9
- rtree: 1.0.1
- safetensors: 0.3.1
- scikit-learn: 1.3.1
- scipy: 1.10.1
- segmentation-models-pytorch: 0.3.3
- send2trash: 1.8.0
- setuptools: 63.4.3
- setuptools-scm: 7.1.0
- shapely: 1.8.4
- six: 1.16.0
- sniffio: 1.3.0
- snowballstemmer: 2.2.0
- snuggs: 1.4.1
- soupsieve: 2.4.1
- sphinx: 5.3.0
- sphinx-design: 0.4.1
- sphinx-rtd-theme: 1.2.2
- sphinxcontrib-applehelp: 1.0.2
- sphinxcontrib-devhelp: 1.0.2
- sphinxcontrib-htmlhelp: 2.0.0
- sphinxcontrib-jquery: 4.1
- sphinxcontrib-jsmath: 1.0.1
- sphinxcontrib-programoutput: 0.15
- sphinxcontrib-qthelp: 1.0.3
- sphinxcontrib-serializinghtml: 1.1.9
- stack-data: 0.6.2
- starlette: 0.27.0
- starsessions: 1.3.0
- sympy: 1.11.1
- tensorboard: 2.13.0
- tensorboard-data-server: 0.7.0
- tensorboard-plugin-wit: 1.8.1
- terminado: 0.17.1
- threadpoolctl: 3.1.0
- timm: 0.9.2
- tinycss2: 1.1.1
- tokenize-rt: 4.2.1
- tomli: 2.0.1
- torch: 2.0.1
- torchmetrics: 1.1.1
- torchvision: 0.15.2
- tornado: 6.2
- tqdm: 4.65.0
- traitlets: 5.9.0
- trove-classifiers: 2023.3.9
- twine: 4.0.2
- typeshed-client: 2.1.0
- typing-extensions: 4.6.3
- tzdata: 2023.3
- uri-template: 1.2.0
- urllib3: 1.26.12
- uvicorn: 0.20.0
- vermin: 1.5.2
- wcwidth: 0.2.5
- webcolors: 1.11.1
- webencodings: 0.5.1
- websocket-client: 1.5.1
- websockets: 10.4
- werkzeug: 2.3.4
- wheel: 0.37.1
- widgetsnbextension: 4.0.3
- xarray: 2023.7.0
- yarl: 1.8.1
- zipfile-deflate64: 0.2.0
- zipp: 3.8.1
- zope.event: 4.6
- zope.interface: 5.4.0
* System:
- OS: Darwin
- architecture:
- 64bit
-
- processor: arm
- python: 3.11.4
- release: 22.6.0
- version: Darwin Kernel Version 22.6.0: Wed Jul 5 22:21:53 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6020

### More info

_No response_

cc @carmocca @mauvilsa

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with torchgeo/datamodules/so2sat.py and torchgeo/datamodules/seco.py, then inspect the linked failing tests and the configs tests/conf/seco_byol_1.yaml and tests/conf/so2sat_s1.yaml. Reproduce the LightningCLI parsing with Lightning 2.0.9 and determine whether kwargs defaults are injected by Lightning or jsonargparse. Done means both datamodules initialize successfully without adding unused or invalid defaults.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.