Add Prometheus Provider
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
Prometheus supports a pushGateway option to push metrics to Prometheus.
This might be useful in cases where metrics needs to be pushed in the ETL stage.
For example, BigQuery has metrics stored in information_schema tables which can be queried using Standard SQL. Its useful to run the Query and store the metrics in prometheus on a periodic basis.
https://github.com/m-lab/prometheus-bigquery-exporter
Suggestion is to add a Prometheus provider first and have a separate PR for BigQueryToPrometheus transfer feature.
### Use case/motivation
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Review the proposed Prometheus provider and the PushGateway use case described in the issue, including the linked prometheus-bigquery-exporter example. Define the provider's scope separately from the proposed BigQueryToPrometheus transfer feature; done means the provider can push metrics to Prometheus through PushGateway.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- prometheus, python
- Domain
- data-engineering, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100