SMTP connection configuration from_email is ignored by email_on_failure and email_on_retry
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Task SDK
### Apache Airflow version
3.2.2
### What happened and how to reproduce it?
Automatic task failure/retry emails sent through the `email_on_failure` and `email_on_retry` task options do not honor the SMTP connection extra `from_email` value.
The SMTP provider's `SmtpNotifier` supports falling back to the SMTP connection extra `from_email` when its own `from_email` argument is `None`, but the task runner always passes a non-`None` fallback value from `[email] from_email` or the hard-coded fallback `airflow@airflow`. As a result, the `SmtpNotifier` connection-extra fallback path is never reached for these automatic task error emails.
### Steps to reproduce
1. Configure the SMTP provider as the email provider and use an SMTP connection for task email notifications.
2. Configure the SMTP connection extra with a sender address, for example:
```json
{
"conn_type": "smtp",
"extra": {
"from_email": "alerts@example.com"
}
}
```
3. Do not set `[email] from_email`.
4. Create a task that uses automatic failure or retry emails, for example:
```python
EmptyOperator(
task_id="failing_task",
email=["operator@example.com"],
email_on_failure=True,
)
```
5. Make the task fail.
6. Airflow trys to send the email from `airflow@airflow`
The reason is that `task-sdk/src/airflow/sdk/execution_time/task_runner.py` constructs the notifier with:
```python
notifier = SmtpNotifier(
to=to_emails,
subject=subject,
html_content=html_content,
from_email=conf.get("email", "from_email", fallback="airflow@airflow"),
)
```
Since `SmtpNotifier.from_email` is never `None`, the following fallback in `providers/smtp/src/airflow/providers/smtp/notifications/smtp.py` is skipped:
```python
if self.from_email is None:
if smtp.from_email is not None:
self.from_email = smtp.from_email
else:
raise ValueError("You should provide `from_email` or define it in the connection")
fields_to_re_render.append("from_email")
```
### What you think should happen instead?
Automatic task failure/retry emails sent using the `SmtpNotifier` should use the most specific configuration available.
I would expect the following outcomes based on the different config fields being set.
1. No `[email] from_email` config set, and `from_email` extra field set for the SMTP connection config: use `from_email` from the SMTP connection config (case from Steps to reproduce, not happening)
----
Other case for completeness reasons
2. No `[email] from_email` config set, and no `from_email` extra field set for the SMTP connection config: use the existing fallback `airflow@airflow` (this is already the case, altough I personally think an error would make more sense)
3. `[email] from_email` config set, and `from_email` extra field not set for the SMTP connection config: use the `[email] from_email` config. (this is already the case)
4. Both `[email] from_email` and the SMTP connection extra `from_email` are set: use `from_email` from the SMTP connection config, because the more specific config should take precedence over the more general config (currently the more general config `[email] from email` is used)
While it is debatable if all of my expected outcomes make sense, I would argue that the current configuration can cause quite a lot of confusion as it did for me, because:
* Airflow 3 uses the `SmtpNotifier` for sending email triggered by `email_on_failure` and `email_on_retry`
* [The SMTP provider changelog for 2.0.0](https://airflow.apache.org/docs/apache-airflow-providers-smtp/stable/changelog.html#id37) explicitely states:
> You can instead use the SMTP connection configuration to set the default values, where you can use:
> the connection extra field `from_email` instead of the configuration `smtp.smtp_mail_from` in `SmtpNotifier`.
### Operating System
_No response_
### Deployment
None
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
I know that both task options `email_on_failure` and `email_on_retry` will be deprecated in Airflow 4 (e.g. see this [PR](https://github.com/apache/airflow/pull/47146)). Nonetheless I thought I should create this issue as Airflow 3 still will be around for quite some time.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start in task-sdk/src/airflow/sdk/execution_time/task_runner.py, where automatic task emails construct SmtpNotifier, then read providers/smtp/src/airflow/providers/smtp/notifications/smtp.py to follow the connection-extra fallback. Verify the precedence cases described in the issue, including both failure and retry emails; done means the SMTP connection from_email is honored when it is the more specific setting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100