[Feature]: We need a memory constrained converting.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 175
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 99
Description
### Feature Description
We need a memory constrained model converting by using auto-round.
For example, we can start the container as follows:
docker run -it --memory="64g" --name bkc2 --runtime=habana -p 30036:30036 -e HABANA_VISIBLE_DEVICES=all -v /dev:/dev --device=/dev:/dev -v $HOME:/workspace -v /data:/data -v /mnt:/mnt -e OMPI_MCA_btl_vader_single_copy_mechanism=none -e PT_HPU_ENABLE_LAZY_COLLECTIVES=true --env http_proxy=$http_proxy --env https_proxy=$https_proxy --env OMPI_ALLOW_RUN_AS_ROOT=1 --env OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 --cap-add=sys_nice --ipc=host --cap-add SYS_PTRACE --cap-add=CAP_IPC_LOCK --ulimit memlock=-1:-1 vault.habana.ai/gaudi-docker/1.23.0/ubuntu24.04/habanalabs/pytorch-installer-2.9.0:latest /bin/bash
But we can convert a model like minimax-2.5 in the container.
### Motivation and Use Case
Currently, many user want to reduce the cost they will install maybe 256G/128G memory. In such machine we need to support the model converting.
### Alternatives Considered
_No response_
### Definition of Done
_No response_
### Additional Context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.