[BUG] Spark Pod Template Resources not getting applied
- Dominant language
- Go
- Stars
- 7.5k
- Forks
- 886
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 120
Description
### Describe the bug
Resources defined in Flyte Pod Templates do not seem to get applied to Flyte Spark Tasks. I am able to submit the following sample code but dont see the resources getting propagated down to the driver or executor pods. Sample Workflow im using:
``` python
from operator import add
import flytekit
from flytekit import ImageSpec, Resources, task, workflow, dynamic, PodTemplate
from flytekitplugins.spark import Spark
from utils import f
from kubernetes.client import (
V1Container,
V1PodSpec,
)
container_image = ImageSpec(
name="sparkfun",
registry="ghcr.io/fiedlernr9",
python_version="3.9",
packages=["flytekitplugins-spark", "kubernetes"],
)
pod_template = PodTemplate(
pod_spec=V1PodSpec(
containers=[
V1Container(
# name="spark-kubernetes-executor",
name="primary",
resources={
"requests": {"cpu": "4" ,"ephemeral-storage": "10Gi"},
"limits": {"cpu": "4", "ephemeral-storage": "10Gi"},
},
)
]
)
)
@task(
task_config=Spark(
# This configuration is applied to the Spark cluster
spark_conf={
"spark.driver.memory": "1000M",
"spark.executor.memory": "4000M",
"spark.executor.cores": "3",
"spark.executor.instances": "5",
"spark.driver.cores": "2",
}
),
pod_template=pod_template,
container_image=container_image,
)
def hello_spark(partitions: int) -> float:
print("Starting Spark with Partitions:! {}".format(partitions))
n = 1 * partitions
sess = flytekit.current_context().spark_session
count = (
sess.sparkContext.parallelize(range(1, n + 1), partitions).map(f).reduce(add)
)
print("Im done")
pi_val = 4.0 * count / n
return pi_val
@task(
container_image=container_image,
pod_template=pod_template,
)
def print_every_time(value_to_print: float):
print("My printed value: {}".format(value_to_print))
@workflow
def wf(partitions: int = 1000) -> float:
pi = hello_spark(partitions=partitions)
print_every_time(value_to_print=pi)
return pi
```
Describing Resources for driver:
```
Limits: │
│ cpu: 1 │
│ memory: 2867Mi │
│ Requests: │
│ cpu: 1 │
│ memory: 2867Mi
```
Describing Resources for executors:
```
Limits: │
│ cpu: 3 │
│ memory: 5600Mi │
│ Requests: │
│ cpu: 3 │
│ memory: 5600Mi
```
### Expected behavior
I would expect the pod template resources to take precedence and see them in the underlying k8s pod definition
### Additional context to reproduce
_No response_
### Screenshots
_No response_
### Are you sure this issue hasn't been raised already?
- [x] Yes
### Have you read the Code of Conduct?
- [x] Yes
Contributor guide
Assessment
This issue has not been assessed yet.