apache / apache/airflow

Passing xcoms as dictionary when using jinja templating to KubernetesPodOperator

Open
#51,296 0 comments 0 reactions 0 assignees View on GitHub
kind:feature needs-triage provider:cncf-kubernetes
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Passing xcoms from upstream tasks to downstream KubernetesPodOperator's env_vars using jinja templating like, `env_vars="{{ task_instance.xcom_pull(task_ids='get_env_vars') }}",` fails as the variables are parsed as strings instead of dictionary.
It would be useful to have a feature that either allows kubernetespodoperator to parse the passed string as a dictionary using another parameter. Alternatively it would be nice to have the xcom pull be rendered as a dictionary by default.

### Use case/motivation

While it is still possible to achieve this using taskflow to load the variables once and use it several times, I would like to avoid having several relationship lines that are automatically created while using the taskflow api. For users like me using variables to store secrets to pass to pods as env, it would be useful to have this functionality to create elegant looking DAGs while at the same time not making unnecessary database calls during parse the DAG, thereby optimising the deployment.

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with KubernetesPodOperator's env_vars handling and the Jinja templating path that processes task_instance.xcom_pull results. Reproduce the string-versus-dictionary behavior, then verify that the chosen feature makes dictionary-valued environment variables usable without the extra TaskFlow relationships described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.