Lightning-AI / Lightning-AI/litgpt

Make `save_hyperparameters()` robust against different CLI entry points

Open
#1,102 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug help wanted
Dominant language
Python
Stars
13.7k
Forks
1.5k
Avg merge
15h 37m
Merged PRs (30d)
1

Description

If you run with

```
litgpt finetune ...
```
and when getting to saving a checkpoint, we hit this line:
https://github.com/Lightning-AI/litgpt/blob/f951f9334610da35c7ecaa7e26e7ba3ac2504dab/litgpt/finetune/lora.py#L193

which reruns the CLI and parses the args that were passed. But this no longer works because it's not the same parser.

```
Saving LoRA weights to 'out/finetune/lora-llama2-7b/step-000200/lit_model.pth.lora'
usage: litgpt [-h] [--config CONFIG] [--print_config[=flags]] [--precision PRECISION] [--quantize QUANTIZE] [--devices DEVICES] [--seed SEED] [--lora_r LORA_R]
[--lora_alpha LORA_ALPHA] [--lora_dropout LORA_DROPOUT] [--lora_query {true,false}] [--lora_key {true,false}] [--lora_value {true,false}]
[--lora_projection {true,false}] [--lora_mlp {true,false}] [--lora_head {true,false}] [--data.help CLASS_PATH_OR_NAME] [--data DATA]
[--checkpoint_dir CHECKPOINT_DIR] [--out_dir OUT_DIR] [--logger_name {wandb,tensorboard,csv}] [--train CONFIG] [--train.save_interval SAVE_INTERVAL]
[--train.log_interval LOG_INTERVAL] [--train.global_batch_size GLOBAL_BATCH_SIZE] [--train.micro_batch_size MICRO_BATCH_SIZE]
[--train.lr_warmup_steps LR_WARMUP_STEPS] [--train.epochs EPOCHS] [--train.max_tokens MAX_TOKENS] [--train.max_steps MAX_STEPS]
[--train.max_seq_length MAX_SEQ_LENGTH] [--train.tie_embeddings {true,false,null}] [--train.learning_rate LEARNING_RATE]
[--train.weight_decay WEIGHT_DECAY] [--train.beta1 BETA1] [--train.beta2 BETA2] [--train.max_norm MAX_NORM] [--train.min_lr MIN_LR] [--eval CONFIG]
[--eval.interval INTERVAL] [--eval.max_new_tokens MAX_NEW_TOKENS] [--eval.max_iters MAX_ITERS]
error: Unrecognized arguments: finetune lora
```

A initial hack to fix this was done in #1103.
Comment by @carmocca
https://github.com/Lightning-AI/litgpt/pull/1103#discussion_r1523182612

> How do you think this could be done? Do we need to choose between jsonargparse.CLI or the CLI in __main__ and then pass the correct one to capture_parser?
>
> We could also simplify this by not having a CLI in the scripts themselves.

We need to make it more robust.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with `litgpt finetune ...` and inspect `litgpt/finetune/lora.py` around line 193, along with the CLI entry points discussed in #1103. Compare how each entry point parses arguments and make checkpoint saving work without reporting `finetune lora` as unrecognized; verify the command completes successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.