SparkKubernetesOperator to support arguments
Open
area:providers
good first issue
kind:documentation
kind:feature
provider:cncf-kubernetes
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
Need an option ,when triggering a dag via airflow web ui, to pass JSON arguments to
SparkKubernetesOperator.
Currently I don't see a way to do it as, SparkKubernetesOperator accepts application.yaml only.
Contributor guide
Research direction
Start by locating SparkKubernetesOperator and the Airflow web UI path for triggering a DAG. Read how the operator currently accepts application.yaml and how trigger parameters are passed. Done means the UI can provide JSON arguments that SparkKubernetesOperator receives and uses as intended.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100