apache / apache/airflow

SMTP connection configuration from_email is ignored by email_on_failure and email_on_retry

Open
#69,262 0 comments 0 reactions 0 assignees View on GitHub
area:core kind:bug needs-triage provider:smtp
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Task SDK

### Apache Airflow version

3.2.2

### What happened and how to reproduce it?

Automatic task failure/retry emails sent through the `email_on_failure` and `email_on_retry` task options do not honor the SMTP connection extra `from_email` value.

The SMTP provider's `SmtpNotifier` supports falling back to the SMTP connection extra `from_email` when its own `from_email` argument is `None`, but the task runner always passes a non-`None` fallback value from `[email] from_email` or the hard-coded fallback `airflow@airflow`. As a result, the `SmtpNotifier` connection-extra fallback path is never reached for these automatic task error emails.

### Steps to reproduce

1. Configure the SMTP provider as the email provider and use an SMTP connection for task email notifications.
2. Configure the SMTP connection extra with a sender address, for example:

```json
{
"conn_type": "smtp",
"extra": {
"from_email": "alerts@example.com"
}
}
```

3. Do not set `[email] from_email`.
4. Create a task that uses automatic failure or retry emails, for example:

```python
EmptyOperator(
task_id="failing_task",
email=["operator@example.com"],
email_on_failure=True,
)
```

5. Make the task fail.
6. Airflow trys to send the email from `airflow@airflow`

The reason is that `task-sdk/src/airflow/sdk/execution_time/task_runner.py` constructs the notifier with:

```python
notifier = SmtpNotifier(
to=to_emails,
subject=subject,
html_content=html_content,
from_email=conf.get("email", "from_email", fallback="airflow@airflow"),
)
```

Since `SmtpNotifier.from_email` is never `None`, the following fallback in `providers/smtp/src/airflow/providers/smtp/notifications/smtp.py` is skipped:

```python
if self.from_email is None:
if smtp.from_email is not None:
self.from_email = smtp.from_email
else:
raise ValueError("You should provide `from_email` or define it in the connection")
fields_to_re_render.append("from_email")
```

### What you think should happen instead?

Automatic task failure/retry emails sent using the `SmtpNotifier` should use the most specific configuration available.

I would expect the following outcomes based on the different config fields being set.

1. No `[email] from_email` config set, and `from_email` extra field set for the SMTP connection config: use `from_email` from the SMTP connection config (case from Steps to reproduce, not happening)
----
Other case for completeness reasons

2. No `[email] from_email` config set, and no `from_email` extra field set for the SMTP connection config: use the existing fallback `airflow@airflow` (this is already the case, altough I personally think an error would make more sense)

3. `[email] from_email` config set, and `from_email` extra field not set for the SMTP connection config: use the `[email] from_email` config. (this is already the case)
4. Both `[email] from_email` and the SMTP connection extra `from_email` are set: use `from_email` from the SMTP connection config, because the more specific config should take precedence over the more general config (currently the more general config `[email] from email` is used)

While it is debatable if all of my expected outcomes make sense, I would argue that the current configuration can cause quite a lot of confusion as it did for me, because:

* Airflow 3 uses the `SmtpNotifier` for sending email triggered by `email_on_failure` and `email_on_retry`
* [The SMTP provider changelog for 2.0.0](https://airflow.apache.org/docs/apache-airflow-providers-smtp/stable/changelog.html#id37) explicitely states:

> You can instead use the SMTP connection configuration to set the default values, where you can use:

> the connection extra field `from_email` instead of the configuration `smtp.smtp_mail_from` in `SmtpNotifier`.

### Operating System

_No response_

### Deployment

None

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

I know that both task options `email_on_failure` and `email_on_retry` will be deprecated in Airflow 4 (e.g. see this [PR](https://github.com/apache/airflow/pull/47146)). Nonetheless I thought I should create this issue as Airflow 3 still will be around for quite some time.

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start in task-sdk/src/airflow/sdk/execution_time/task_runner.py, where automatic task emails construct SmtpNotifier, then read providers/smtp/src/airflow/providers/smtp/notifications/smtp.py to follow the connection-extra fallback. Verify the precedence cases described in the issue, including both failure and retry emails; done means the SMTP connection from_email is honored when it is the more specific setting.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.