cityofaustin / cityofaustin/atd-data-tech
Improve Airflow Slack notifications
- Dominant language
- No language data
- Stars
- 33
- Forks
- 3
- Avg merge
- 5d 14h
- Merged PRs (30d)
- 2
Description
We talked about improving the Airflow Slack notification behavior in our retro recently - [What Could Be Improved](https://docs.google.com/document/d/1P1Gf_y29AcKrNPLcIMUkLfdEEPW0efK7whuihtcuUrk/edit?tab=t.0#heading=h.28pzdeoqnsb0).
Pain points:
- Too much noise, hard to tell what needs action
- Can't tell if a DAG runs frequently and will heal itself if it comes out of failure
- VPN is speed bump to checking error messages
Ideas:
- Utilizing different alert levels https://github.com/cityofaustin/atd-airflow/blob/production/dags/utils/slack_operator.py
- We have an existing critical level which mentions `@channel` - CRIS DAG for example
- [x] Showing the schedule (could be human readable? ~https://github.com/bradymholt/cron-expression-descriptor~ here is the python port https://github.com/Salamek/cron-descriptor) - https://github.com/cityofaustin/atd-data-tech/issues/20070
- Global rule to differentiate between repeated failures or initial failure
- Enhance alert messages with metadata available through Airflow metadata (or DB or API)
- https://github.com/cityofaustin/atd-data-tech/issues/20645
- https://github.com/cityofaustin/atd-airflow/pull/270
- Make dashboard accessible over the public internet
- Extend Airflow DAG object to include `importance` messages or custom icons that contain specific critical info about a DAG to help developers address failures. See [this Slack alert](https://austininnovation.slack.com/archives/C013G4RNJ01/p1733863516073849) for an example https://github.com/cityofaustin/atd-data-tech/issues/20645
- Example of mentions in DAG byline https://github.com/cityofaustin/atd-airflow/pull/271
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.