Airflow Provider for OpenTelemetry
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Description
Opentelemetry (https://opentelemetry.io/) is quickly becoming the standard of sending telemetry data such as metrics, logs, and traces) to any targets supporting its protocol. Airflow as a popular choice of data pipelines need to send opentelemetry telemetry to any targets during its operation to make the pipeline be observed by many monitoring/observability tools currently available. This feature for OpenTelemetry will contain plugins, listeners, and hooks to make it easy for users to send out their own telemetry data via OpenTelemetry standards.
### Use case/motivation
By providing 'opentelemetry provider' for airflow, user can easily use the provider that contains :
1. listeners - to listen for task runs and dag runs, and emit them out as opentelemetry traces to configured OTEL endpoint.
2. provider opentelemetry hooks for users sending out traces, and logs using opentelemetry within their DAG source codes
## Motivation for this feature.
[AIP-49](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow) is finally released and became part of native Airflow support. With the new OTEL tracing capability in place, when enabled, users should be able to monitor:
- The performance and detailed trace of events happening around DAG run. Users would be able to note when each of the tasks inside the DAG run happened, and what was their outcome. Additionally, these information is now capable of being sent into OTLP compatible endpoint for collecting, storing, and analyzing in the compatible opentelemetry backend of choice.
- Not only that, there are other traces emitted from Airflow processes such as scheduler, triggerer, and executor that will emit trace on when and how long each heartbeat had taken place, how long did the dag processing took place, whether there were any errors, and the interaction between each task instances and executors linked together, such that users would be able to identify how the DAG runs were queued, and executed in much more detail.
### But what about custom spans and attributes?
DAG run now can be expressed in traces which would look like the following:
These relationship between when a certain task ran at which time and order, and should be good enough for most of the observing how well the DAG run went.
However, this does not capture per task specific details that can describe additional information such as
- What were the business context and details specific to certain runs?
- Any additional information that could not be collected as part of the basic Run itself, such as
- data quality related information that user may want to instrument
- any execution or runtime information specific to certain operators or external systems
- any additional logs in which can be part of span events
```python
from otel_hook import OtelHook
...
def setup(**dag_context):
with otel_hook.start_as_current_span(name="do_setup", dag_context=dag_context) as s:
s.set_attribute("data quality", "fair")
s.set_attribute("description", "You can add attributes in otel hook to have business or data specific details on top of existing task instnace span.")
with otel_hook.start_as_current_span(name="do_sleep") as ss:
ss.set_attribute("sleep for", "one second")
time.sleep(1)
# simple setup operator in python
t0 = PythonOperator(
task_id="setup",
python_callable=setup
)
```
Providing such means for the users to instrument any DAG run specific spans or contexts and make them as part of the task instance would tremendously help to gain deeper insights. Imagine if the user can easily add their own Spans inside their DAG file to record certain activities happening:
OTEL provider will have OTEL hook that can help you to do this. Using the hook, you may be able to create active span and attach it to the current running DAG's task instance, conveniently.
### Supporting lower version of Airflow to emit DAG run trace
Another feature for the OpenTelemetry provider is providing ways to capture DAG runs in case the Airflow version is lower (v2.10.0) than the Airflow that has OTEL trace enabled. In that case, with the limited capability, OTEL provider can generate traces coming from the task instances, that can complement Airflows that either have lower version or have the OTEL capability disabled. In that case, provider can automatically detect and start generating its own traces regardless.
This can also be combined with user's custom instrumentation so that the instrumentation can still be used, whether the Airflow is enabled with OTEL, or airflow is using OTEL provider.
- OTEL provider will implement an event listener.
- this event listener will only work when it is enabled via configuration
- If the even listener detects the airflow version is >= 2.10.0, then it will try to check whether OTEL instrumentation is enabled. If so, it will use its configuration (endpoint) to emit traces. If not, it will default to the configuration provided by its connection info.
> However, event listener can only detect and collect whatever information is available via DagRun and TaskInstance, and therefore will not be able to collect all the traces (e.g. scheduler job) that are available from OTEL traces for Airflow.
This feature could also be useful if user is unable to upgrade into Airflow v2.10, and still want to leverage the OTEL tracing.
### Related issues
None
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by reading AIP-49 and the requested provider behavior in this issue. Define the scope for listeners, OpenTelemetry hooks, configuration, and compatibility with Airflow versions before implementation. Done means the provider can emit the described telemetry and support user instrumentation, with the required behavior verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100