intel / intel/auto-round

[Feature]: We need a memory constrained converting.

Open
#1,702 1 comment 0 reactions 1 assignee Assigned to @thuang6 View on GitHub
enhancement
Dominant language
Python
Stars
1.6k
Forks
175
Avg merge
1d 18h
Merged PRs (30d)
99

Description

### Feature Description

We need a memory constrained model converting by using auto-round.

For example, we can start the container as follows:
docker run -it --memory="64g" --name bkc2 --runtime=habana -p 30036:30036 -e HABANA_VISIBLE_DEVICES=all -v /dev:/dev --device=/dev:/dev -v $HOME:/workspace -v /data:/data -v /mnt:/mnt -e OMPI_MCA_btl_vader_single_copy_mechanism=none -e PT_HPU_ENABLE_LAZY_COLLECTIVES=true --env http_proxy=$http_proxy --env https_proxy=$https_proxy --env OMPI_ALLOW_RUN_AS_ROOT=1 --env OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 --cap-add=sys_nice --ipc=host --cap-add SYS_PTRACE --cap-add=CAP_IPC_LOCK --ulimit memlock=-1:-1 vault.habana.ai/gaudi-docker/1.23.0/ubuntu24.04/habanalabs/pytorch-installer-2.9.0:latest /bin/bash

But we can convert a model like minimax-2.5 in the container.

### Motivation and Use Case

Currently, many user want to reduce the cost they will install maybe 256G/128G memory. In such machine we need to support the model converting.

### Alternatives Considered

_No response_

### Definition of Done

_No response_

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.