deepspeedai / deepspeedai/DeepSpeedExamples

When running Stage-3 scripts with enable_hybrid_engine encountered errors

Open
#373 9 comments 4 reactions 2 assignees View on GitHub

@cmikeh2 is already working on this.

Since Apr 24, 2023.

deespeed chat hybrid engine
Dominant language
Python
Stars
6.8k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
1

Description

I was using script from step3_rlhf_finetuning/training_scripts/single_node/run_6.7b.sh, I met some errors.
I used 7B Llama models as actor and critic respectively and set enable_hybrid_engine argument, I got errors like below:

│ /root/miniconda3/envs/coati/lib/python3.10/site-packages/deepspeed/runtime/hybrid_engine.py:398 │
│ in step │
│ │
│ 395 │ │
│ 396 │ def step(self, lr_kwargs=None): │
│ 397 │ │ super().step(lr_kwargs=lr_kwargs) │
│ ❱ 398 │ │ if(self._inference_containers[0].module.attention.attn_qkvw is not None and \ │
│ 399 │ │ │ self._inference_containers[0].q_k_v is not None): │
│ 400 │ │ │ for inference_container in self._inference_containers: │
│ 401 │ │ │ │ inference_container.reset_qkv() │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
IndexError: list index out of range

How can I solve this issue?
Thx : )

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.