apache / apache/airflow

Add Prometheus Provider

Open
#20,053 8 comments 0 reactions 0 assignees View on GitHub
area:providers good first issue kind:new provider request
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Prometheus supports a pushGateway option to push metrics to Prometheus.
This might be useful in cases where metrics needs to be pushed in the ETL stage.

For example, BigQuery has metrics stored in information_schema tables which can be queried using Standard SQL. Its useful to run the Query and store the metrics in prometheus on a periodic basis.

https://github.com/m-lab/prometheus-bigquery-exporter

Suggestion is to add a Prometheus provider first and have a separate PR for BigQueryToPrometheus transfer feature.
### Use case/motivation

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Review the proposed Prometheus provider and the PushGateway use case described in the issue, including the linked prometheus-bigquery-exporter example. Define the provider's scope separately from the proposed BigQueryToPrometheus transfer feature; done means the provider can push metrics to Prometheus through PushGateway.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus, python
Domain
data-engineering, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.