linkedin / linkedin/dr-elephant
Airflow context not showing up in javaExtraOptions
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.4k
- Forks
- 839
- PR merge metrics
- No merged PRs in 30d
Description
Hello!
I'm having some trouble linking jobs scheduled in Airflow to Dr. Elephant. I've got a running version of Dr. Elephant on our Spark cluster, running YARN mode, with Airflow scheduling jobs. I have, however, been unable to get a connection between tasks/DAGs to show up in Dr. E. After some investigation, the Airflow context information is not being recorded in the javaExtraOptions portion of the config. Has anyone else had this problem?
I'm using the BashOperator to execute spark-submit jobs through bash. I can verify that Airflow is recording the airflow context vars into the BashOperators environment, but they never show up in the location where Dr. E is looking for them.
Anyone advice or other suggestions on how to resolve this would be helpful. Thanks!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how BashOperator passes Airflow context variables to the spark-submit command and where Dr. Elephant reads javaExtraOptions. Reproduce the YARN-mode job, then verify that the Airflow task or DAG context is recorded where Dr. Elephant expects it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100