CarperAI / CarperAI/trlx

Increasing max new tokens for generation arguments lead to errors

Open
#553 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
4.8k
Forks
487
PR merge metrics
No merged PRs in 30d

Description

### 🐛 Describe the bug

Here's my `TrainConfig`:
```python
default_config = TRLConfig(
train=TrainConfig(
seq_length=512,
epochs=10000,
total_steps=10000,
batch_size=8,
checkpoint_interval=10000,
eval_interval=500,
pipeline="PromptPipeline",
trainer="AcceleratePPOTrainer",
checkpoint_dir="checkpoints/ppo_hh",
),
model=ModelConfig(model_path="tiiuae/falcon-7b-instruct", num_layers_unfrozen=2),
tokenizer=TokenizerConfig(tokenizer_path="tiiuae/falcon-7b-instruct", truncation_side="left"),
optimizer=OptimizerConfig(name="adamw", kwargs=dict(lr=1e-6, betas=(0.9, 0.95), eps=1.0e-8, weight_decay=1.0e-6)),
scheduler=SchedulerConfig(name="cosine_annealing", kwargs=dict(T_max=10000, eta_min=1e-6)),
method=PPOConfig(
name="PPOConfig",
num_rollouts=64,
chunk_size=4,
ppo_epochs=4,
init_kl_coef=0.05,
target=6,
horizon=10000,
gamma=1,
lam=0.95,
cliprange=0.2,
cliprange_value=0.2,
vf_coef=1,
scale_reward="running",
ref_mean=None,
ref_std=None,
cliprange_reward=10,
gen_kwargs=dict(
max_new_tokens=128,
top_k=0,
top_p=1.0,
do_sample=True,
),
),
)
```

Simply changing `max_new_tokens` from 128 to 256 leads to an error:
```
File "/home/ec2-user/trlx/examples/hh/ppo_hh.py", line 263, in
main(hparams)
File "/home/ec2-user/trlx/examples/hh/ppo_hh.py", line 252, in main
trlx.train(
File "/home/ec2-user/trlx/trlx/trlx.py", line 129, in train
trainer.learn()
File "/home/ec2-user/trlx/trlx/trainer/accelerate_base_trainer.py", line 521, in learn
self.prepare_learning()
File "/home/ec2-user/trlx/trlx/trainer/accelerate_ppo_trainer.py", line 234, in prepare_learning
self.make_experience(self.config.method.num_rollouts)
File "/home/ec2-user/trlx/trlx/trainer/accelerate_ppo_trainer.py", line 283, in make_experience
samples = self.generate(batch["input_ids"], batch["attention_mask"])
File "/home/ec2-user/trlx/trlx/trainer/accelerate_base_trainer.py", line 263, in generate
return self.accelerator.unwrap_model(self.model).generate(
Traceback (most recent call last):
File "/home/ec2-user/trlx/trlx/models/modeling_ppo.py", line 353, in generate
return self.base_model.generate(*args, **kwargs)
File "/home/ec2-user/trlx/examples/hh/ppo_hh.py", line 263, in
File "/opt/conda/envs/vllm/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/opt/conda/envs/vllm/lib/python3.9/site-packages/transformers/generation/utils.py", line 1476, in generate
and torch.sum(inputs_tensor[:, -1] == generation_config.pad_token_id) > 0
main(hparams)
File "/home/ec2-user/trlx/examples/hh/ppo_hh.py", line 252, in main
IndexError: index -1 is out of bounds for dimension 1 with size 0
```

Any help is appreciated!

### Which trlX version are you using?

0.7.0

### Additional system and package information

Python=3.9

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.