huggingface / huggingface/diffusers

How about forcing the first and last block on device when groupoffloading is used?

Open
#11,966 16 comments 0 reactions 1 assignee Claimed by @a-r-r-o-w View on GitHub
contributions-welcome group-offloading stale
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

**Is your feature request related to a problem? Please describe.**
When group offloading is enabled, the offload and onload cannot be streamed between steps and this is really a big time comsuming problem.

**Describe the solution you'd like.**
Is it possible to add an option that could make the first and last block forced on device to avoid offload and onload?

@a-r-r-o-w Could you please give some help? Thanks so much.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.