OpenMOSS / OpenMOSS/MOSS

ZeRORuntimeException: You are using ZeRO-Offload with a client provided optimizer

Open
#272 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
12.3k
Forks
1.1k
PR merge metrics
No merged PRs in 30d

Description

显卡配置:2张 V100 32G (共四张,有两张别人占用中,用完后可实现利用4卡V100)
按照默认accelerate配置报错:cuda out of memory,观察发现默认配置中 offload_optimizer_device 和 offload_param_device 参数均为none,后按照accelerate教程,将这两个参数均改成 cpu 报错:

image

accelerate 配置如下:

command_file: null
commands: null
compute_environment: LOCAL_MACHINE
deepspeed_config:
gradient_accumulation_steps: 1
gradient_clipping: 1.0
offload_optimizer_device: cpu
offload_param_device: cpu
zero3_save_16bit_model: true
zero_stage: 3
zero3_init_flag: true
zero_force_ds_cpu_optimizer: False
distributed_type: DEEPSPEED
downcast_bf16: 'no'
dynamo_backend: 'NO'
fsdp_config: {}
gpu_ids: null
machine_rank: 0
main_process_ip: null
main_process_port: null
main_training_function: main
megatron_lm_config: {}
mixed_precision: fp16
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
tpu_name: null
tpu_zone: null
use_cpu: false

和别人沟通后得知8*A100(40G)可按照官方的脚本直接训练,不用修改cpu offload参数,受限于机器资源,怎样能实现cpu offload 且不报错呢?需要配置deepspeed 配置文件(.json)吗?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the provided Accelerate and DeepSpeed configuration, then reproduce the ZeRORuntimeException with CPU offload enabled on the stated GPU setup. Done means identifying a supported configuration or documenting that CPU offload cannot work with this training path without changing the optimizer setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.