llava-llama3-8b 微调过程中 loss nan
Open
- Dominant language
- Python
- Stars
- 5.2k
- Forks
- 448
- Avg merge
- 3d 15h
- Merged PRs (30d)
- 26
Description
微调llava-llama3-8b的时候 从几个step后就开始loss=nan了 这个可能是什么原因呢?我看github issue也有人遇到类似问题 官方回复是改lr 我现在设置的
```
# Scheduler & Optimizer
batch_size = 4 # per_device
accumulative_counts = 32*4
dataloader_num_workers = 32
max_epochs = 1
optim_type = AdamW
lr = 2e-6
param_scheduler = [
dict(
type=LinearLR,
start_factor=1e-5,
by_epoch=True,
begin=0,
end=warmup_ratio * max_epochs,
convert_to_iter_based=True),
dict(
type=CosineAnnealingLR,
eta_min=0.0,
by_epoch=True,
begin=warmup_ratio * max_epochs,
end=max_epochs,
convert_to_iter_based=True)
]
```
Contributor guide
Assessment
This issue has not been assessed yet.