Errors when try to save checkpoints during the full fine-tuning

Open

#1,764 opened on Oct 1, 2024 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

help wanted question

Dominant language: Python
Stars: 13.7k
Forks: 1.5k
Avg merge: 15h 37m
Merged PRs (30d): 1

Description

When I try to save checkpoints during full fine-tune, errors happen when running `save_hyperparameters(setup, check_file.parent)` this line. The detailed errors are shown below:
```
[rank0]: save_hyperparameters(setup, checkpoint_file.parent)
[rank0]: File "litgpt/litgpt/utils.py", line 487, in save_hyperparameters
[rank0]: parser = capture_parser(lambda: CLI(function))
[rank0]: File ".conda/envs/gpt/lib/python3.10/site-packages/jsonargparse/_util.py", line 109, in capture_parser
[rank0]: function(*args, **kwargs)
[rank0]: File "litgpt/litgpt/utils.py", line 487, in
[rank0]: parser = capture_parser(lambda: CLI(function))
[rank0]: File "litgpt/litgpt/utils.py", line 449, in CLI
[rank0]: return CLI(*args, **kwargs)
[rank0]: File ".conda/envs/gpt/lib/python3.10/site-packages/jsonargparse/_cli.py", line 88, in CLI
[rank0]: _add_component_to_parser(components, parser, as_positional, fail_untyped, config_help)
[rank0]: File ".conda/envs/gpt/lib/python3.10/site-packages/jsonargparse/_cli.py", line 187, in _add_component_to_parser
[rank0]: added_args = parser.add_function_arguments(component, as_group=False, **kwargs)
[rank0]: File ".conda/envs/gpt/lib/python3.10/site-packages/jsonargparse/_signatures.py", line 209, in add_function_arguments
[rank0]: return self._add_signature_arguments(
[rank0]: File ".conda/envs/gpt/lib/python3.10/site-packages/jsonargparse/_signatures.py", line 289, in _add_signature_arguments
[rank0]: self._add_signature_parameter(
[rank0]: File ".conda/envs/gpt/lib/python3.10/site-packages/jsonargparse/_signatures.py", line 418, in _add_signature_parameter
[rank0]: raise ValueError(
[rank0]: ValueError: With fail_untyped=True, all mandatory parameters must have a supported type. Parameter 'checkpoint_dir' from '__main__.setup' does not specify a type.
```

Contributor guide

Open the contributing guide

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Research direction

Start with litgpt/litgpt/utils.py around line 487 and inspect how save_hyperparameters captures the __main__.setup CLI signature. Reproduce the full fine-tuning checkpoint-save workflow and verify that saving completes without the jsonargparse ValueError for checkpoint_dir.

Written by the indexing model from the issue text.

Assessment

Tech stack: python
Domain: machine-learning
Issue type: Bug
Difficulty: 3/5
Estimated time: 1-2 days
Activity status: Stale
Clarity: Needs clarification
Newbie friendliness: 25/100