flyteorg / flyteorg/flyte

[BUG] Spark Pod Template Resources not getting applied

Open
#6,348 22 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
7.5k
Forks
886
Avg merge
1d 12h
Merged PRs (30d)
120

Description

### Describe the bug

Resources defined in Flyte Pod Templates do not seem to get applied to Flyte Spark Tasks. I am able to submit the following sample code but dont see the resources getting propagated down to the driver or executor pods. Sample Workflow im using:

``` python
from operator import add

import flytekit
from flytekit import ImageSpec, Resources, task, workflow, dynamic, PodTemplate
from flytekitplugins.spark import Spark
from utils import f
from kubernetes.client import (
V1Container,
V1PodSpec,
)

container_image = ImageSpec(
name="sparkfun",
registry="ghcr.io/fiedlernr9",
python_version="3.9",
packages=["flytekitplugins-spark", "kubernetes"],
)

pod_template = PodTemplate(
pod_spec=V1PodSpec(
containers=[
V1Container(
# name="spark-kubernetes-executor",
name="primary",
resources={
"requests": {"cpu": "4" ,"ephemeral-storage": "10Gi"},
"limits": {"cpu": "4", "ephemeral-storage": "10Gi"},
},
)
]
)
)

@task(
task_config=Spark(
# This configuration is applied to the Spark cluster
spark_conf={
"spark.driver.memory": "1000M",
"spark.executor.memory": "4000M",
"spark.executor.cores": "3",
"spark.executor.instances": "5",
"spark.driver.cores": "2",
}
),
pod_template=pod_template,
container_image=container_image,
)
def hello_spark(partitions: int) -> float:
print("Starting Spark with Partitions:! {}".format(partitions))

n = 1 * partitions
sess = flytekit.current_context().spark_session
count = (
sess.sparkContext.parallelize(range(1, n + 1), partitions).map(f).reduce(add)
)
print("Im done")
pi_val = 4.0 * count / n
return pi_val

@task(
container_image=container_image,
pod_template=pod_template,
)
def print_every_time(value_to_print: float):
print("My printed value: {}".format(value_to_print))

@workflow
def wf(partitions: int = 1000) -> float:
pi = hello_spark(partitions=partitions)
print_every_time(value_to_print=pi)
return pi
```

Describing Resources for driver:
```
Limits: │
│ cpu: 1 │
│ memory: 2867Mi │
│ Requests: │
│ cpu: 1 │
│ memory: 2867Mi
```
Describing Resources for executors:

```
Limits: │
│ cpu: 3 │
│ memory: 5600Mi │
│ Requests: │
│ cpu: 3 │
│ memory: 5600Mi
```

### Expected behavior

I would expect the pod template resources to take precedence and see them in the underlying k8s pod definition

### Additional context to reproduce

_No response_

### Screenshots

_No response_

### Are you sure this issue hasn't been raised already?

- [x] Yes

### Have you read the Code of Conduct?

- [x] Yes

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.