Support node replicas in `/api/v2/monitor/health` response
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Description
Modify the `/api/v2/monitor/health` response to surface information for the state of any horizontally scaled / replicated components. Given that all the components described in the current response are horizontally scalable, It would be much more useful to return a list of state payloads for each component. For example, in a high-availability scheduler setup running 2 schedulers, it would be useful to see the status information for both those two schedulers in the response.
A very rough outline of what the proposed API response _might_ look like is below:
```json
{
"metadatabase": {
"status": "string"
},
"scheduler": [
{
"hostname": "scheduler-1",
"status": "healthy",
"latest_scheduler_heartbeat": "scheduler-1 latest heartbeat here"
},
{
"hostname": "scheduler-2",
"status": "healthy",
"latest_scheduler_heartbeat": "scheduler-2 latest heartbeat here"
}
],
"triggerer": [
...
],
"dag_processor": [
...
]
}
```
Concretely, I believe the most noteworthy changes necessary for this would be:
1. Adding a `hostname` field to the `(Scheduler|Triggerer|DagProcessor)InfoResponse` models in `airflow.api_fastapi.core_api.datamodels.monitor`
2. Changing the fields of `HealthInfoResponse` model to be a list of the corresponding per-field models.
3. Adding an index on the `hostname` column in the `jobs` table (this particular change would benefit a number of other things unrelated to this ticket).
### Use case/motivation
Currently, as of [version 3.1.7](https://airflow.apache.org/docs/apache-airflow/3.1.7/stable-rest-api-ref.html#operation/get_health), responses from the `/api/v2/monitor/health` endpoint offer a single `status` and `latest_*_heartbeat` field per Airflow component. This is fine for any Airflow deployment which has a single scheduler, a single triggerer, and so on. However, the information from this endpoint does not provide reliable insight into the system's health if one or more of those components are horizontally scaled.
### Related issues
Tangentially, the feature proposed here could make it easier to implement https://github.com/apache/airflow/issues/17191, but to be clear that ticket describes a different feature.
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Assessment
This issue has not been assessed yet.