flyteorg / flyteorg/flyte

[Core feature] Greater flexibility with GPUs accelerated workloads

Open
#6,743 2 comments 0 reactions 1 assignee Claimed by @Sovietaced View on GitHub
enhancement
Dominant language
Go
Stars
7.5k
Forks
886
Avg merge
1d 12h
Merged PRs (30d)
120

Description

### Motivation: Why do you think this is important?

Right now the configuration around GPU accelerated workloads is quite rigid. Flyte propeller only allows for a global GPU resource name to be configured which makes impossible to use Flyte with data planes that have heterogenous GPU resource vendors/names.

Additionally, it is assumed that all GPU accelerated compute nodes will require a singular node selector label or taint key/value. In our environment our compute nodes contain multiple taints.

We have worked around some of these issues using pod templates but we hit a blocker when we needed to support fractionalized GPUs with `nvidia.com/gpu.shared`.

### Goal: What should the final outcome look like, ideally?

A more flexible configuration for GPU accelerated workloads where custom resource names, node selectors, and tolerations can be configured.

### Describe alternatives you've considered

None

### Propose: Link/Inline OR Additional context

_No response_

### Are you sure this issue hasn't been raised already?

- [x] Yes

### Have you read the Code of Conduct?

- [x] Yes

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.