huggingface / huggingface/accelerate

[feature-request] Add OpenVINO as an inference-only backend

Open
#2,306 2 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
Python
Stars
9.9k
Forks
1.5k
Avg merge
5d 2h
Merged PRs (30d)
27

Description

**What is OpenVINO**?
[OpenVINO](https://github.com/openvinotoolkit/openvino) enables inference optimizations for various devices. Huggingface already provides inference optimizations with [optimum-intel](https://github.com/huggingface/optimum-intel).

**Why support it**?
I am using `accelerate` mainly to write device agnostic code in combination with various HF libraries. `Accelerator.device()` being the most useful feature next to `accelerate launch` for example with [evaluation-harness](https://github.com/bigcode-project/bigcode-evaluation-harness).
There are [reports](https://github.com/vladmandic/automatic/blob/master/README.md) of OpenVINO working with AMD GPUs on Windows, and it's also much simpler than installing IPEX for GPU inference with Intel GPU.
Models in OpenVINO Intermedia Representation can be supported (including various quantizations)

**Limitations**
OpenVINO is inference only.
It's main target is edge devices and vision model.

even if this is not a good fit, I simply wanted to have the idea out there for others to find. Accelerate already supports onnxruntime which does a lot of similar things, and can also use openvino as an execution provider on supported devices (namely Intel NPU).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.