influxdata / influxdata/kapacitor
smtp service/gomail fails to send succeeding alert using AWS SES due to the default idleTimeout
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
Hi, I'd like to report a possible bug wherein the default idleTimeout (30s) causes Kapacitor to fail in sending email alerts.
Environment:
X-Kapacitor-Version: 1.5.4
inside a docker container
CentOS Linux release 7.6.1810 (Core)
How to replicate:
1. Using the ff config (sanitized).
```
curl -s "localhost:9092/kapacitor/v1/config/smtp/" | jq .
{
"link": {
"rel": "self",
"href": "/kapacitor/v1/config/smtp/"
},
"options": {
"enabled": true,
"from": "kapacitor@email.com",
"global": true,
"host": "email-smtp.eu-west-1.amazonaws.com",
"idle-timeout": "30s",
"no-verify": false,
"password": true,
"port": 465,
"state-changes-only": false,
"to": [
"nms@email.com"
],
"username": "redacted"
},
"redacted": [
"password"
]
}
```
2. Send a couple of test message.
```
curl -X POST "localhost:9092/kapacitor/v1/service-tests/smtp" -d '{"body": "hi", "subject": "hi", "to": ["my@email.com"]}';
sleep 29;
curl -X POST "localhost:9092/kapacitor/v1/service-tests/smtp" -d '{"body": "hi", "subject": "hi", "to": ["my@email.com"]}';
sleep 5;
curl -X POST "localhost:9092/kapacitor/v1/service-tests/smtp" -d '{"body": "hi", "subject": "hi", "to": ["my@email.com"]}';
```
3. The result will be the first attempt to send an email will be successful and second and third attempt will fail w/ the following errors in the logs.
```
ts=2020-06-24T11:21:00.129Z lvl=error msg="error sending" service=smtp err="gomail: could not send email 1: 451 4.4.2 Timeout waiting for data from client."
ts=2020-06-24T11:21:05.141Z lvl=error msg="error sending" service=smtp err="gomail: could not send email 1: write tcp 172.17.0.2:47642->52.213.105.200:587: write: broken pipe"
ts=2020-06-24T11:21:35.141Z lvl=error msg="error closing connection to SMTP server" service=smtp err="write tcp 172.17.0.2:47642->52.213.105.200:587: write: broken pipe"
```
I notice that SES does close idle connection but not sure if Kapacitor idleTimeout is supposed to handle that.
```
time telnet email-smtp.eu-west-1.amazonaws.com 465
Trying 52.208.176.178...
Connected to email-smtp.eu-west-1.amazonaws.com.
Escape character is '^]'.
Connection closed by foreign host.
real 0m31.294s
```
Currently the only way to reliably send multiple email is to override the idleTimeout to 0s.
`curl -X POST "localhost:9092/kapacitor/v1/config/smtp/" -d '{"set":{"idle-timeout": "0s"}}'`
Contributor guide
Research direction
Start with the SMTP configuration endpoint at /kapacitor/v1/config/smtp/ and the service test entry point at /kapacitor/v1/service-tests/smtp. Reproduce the three sends with the 30s idle-timeout and compare them with idle-timeout set to 0s, then trace how the SMTP connection handles an SES-closed idle connection. Done means repeated sends succeed without broken-pipe or timeout errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100