cityofaustin / cityofaustin/atd-data-tech

Improve Airflow Slack notifications

Open
#20,008 3 comments 0 reactions 1 assignee Claimed by @mddilley View on GitHub
Epic Team: Dev Type: DevOps Type: Enhancement Workgroup: DTS
Dominant language
No language data
Stars
33
Forks
3
Avg merge
5d 14h
Merged PRs (30d)
2

Description

We talked about improving the Airflow Slack notification behavior in our retro recently - [What Could Be Improved](https://docs.google.com/document/d/1P1Gf_y29AcKrNPLcIMUkLfdEEPW0efK7whuihtcuUrk/edit?tab=t.0#heading=h.28pzdeoqnsb0).

Pain points:
- Too much noise, hard to tell what needs action
- Can't tell if a DAG runs frequently and will heal itself if it comes out of failure
- VPN is speed bump to checking error messages

Ideas:

- Utilizing different alert levels https://github.com/cityofaustin/atd-airflow/blob/production/dags/utils/slack_operator.py
- We have an existing critical level which mentions `@channel` - CRIS DAG for example
- [x] Showing the schedule (could be human readable? ~https://github.com/bradymholt/cron-expression-descriptor~ here is the python port https://github.com/Salamek/cron-descriptor) - https://github.com/cityofaustin/atd-data-tech/issues/20070
- Global rule to differentiate between repeated failures or initial failure
- Enhance alert messages with metadata available through Airflow metadata (or DB or API)
- https://github.com/cityofaustin/atd-data-tech/issues/20645
- https://github.com/cityofaustin/atd-airflow/pull/270
- Make dashboard accessible over the public internet
- Extend Airflow DAG object to include `importance` messages or custom icons that contain specific critical info about a DAG to help developers address failures. See [this Slack alert](https://austininnovation.slack.com/archives/C013G4RNJ01/p1733863516073849) for an example https://github.com/cityofaustin/atd-data-tech/issues/20645
- Example of mentions in DAG byline https://github.com/cityofaustin/atd-airflow/pull/271

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.