Atlas Backend module improvements
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
**Description**
There is a requirement to send lineage info from Apache Airflow to Apache Atlas as a part of data lineage implementation. However, current module which was represented in 1.10 stable version has a few problems:
1. It creates new Atlas Operator entity for each DagRun.
2. It's impossible to control missing entities creation using configs.
3. It fails operator if lineage sending was not succesful.
4. HTTP timeout can't be configured.
5. Current Atlas type definition has a small set of attributes.
6. Errors in class wrappers for Atlas types.
**Use case / motivation**
As a part of analytic data platform, auto import of data lineage is needed and the most part of data lineage should be send by Airflow in auto mode. Our module uses old Atlas backend module as a base, but contains fixes and improvements. What was fixed:
1. Creation of Atlas entity of Airflow operator doesn't use execution date anymore.
2. Added config property for enabling/disabling missing inlets and outlets entities creation.
3. Added config property for enabling/disabling operator failure if lineage sending was unsuccessful.
4. Added config property for the Atlas timeout.
5. Added "template_fields" into Airflow operator typedef and added additional config property for setting any additional operator attributes
6. Fixed DataSet class wrapper, added abstract types for file and JDBC source
7. Added utils methods for correct inlets and outlets objects generating.
**Related Issues**
AIRFLOW-5912
Contributor guide
Research direction
Start by reviewing the existing Atlas backend module from the Airflow 1.10 stable version and the related AIRFLOW-5912 issue. Done means addressing the seven listed improvements, including configurable entity creation, failure handling, timeout, Atlas attributes and type wrappers, and lineage object generation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100