InternLM / InternLM/lmdeploy

[Feature] Support deploy model on specific device id

Open
#2,925 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

### Motivation

I notice that if I want to deploy a LLM on a GPU device with specific device id, the only way I can do is `export CUDA_VISIBLE_DEVICES=xxx`.
Is there any other way to set the specific device id of lmdeploy instance? that the tensor from another device id can be addressed on this device, rather than be limited by the id listed in CUDA_VISIBLE_DEVICES?

### Related resources

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.