Nutmeg: Sending email reports complete success, but I can see from logs it partially failed

Issue is described here: https://discuss.openedx.org/t/sending-email-reports-complete-success-but-i-can-see-from-logs-it-partially-failed/9537

In a nutshell, if the smtplib.py receives a SMTP disconnect, it errors out, but then goes on to continue to incorrectly say that it succeeded, and report that incorrect success in the Instructor UI.

lms-worker_1                 | 2023-03-10 14:19:17,590 INFO 11 [edx.celery.task] [user None] [ip None] tasks.py:564 - BulkEmail ==> Task: a649930b-8a85-4ef8-a740-c006f6be6914, SubTask: 97285059-a3f0-409a-b3c6-b79ed9ce0a57, EmailId: 53, Recipient num: 91/119, Recipient UserId: 937
lms-worker_1                 | 2023-03-10 14:19:17,677 ERROR 11 [edx_ace.channel.django_email] [user 937] [ip 127.0.0.1] django_email.py:70 - Connection unexpectedly closed
lms-worker_1                 | Traceback (most recent call last):
lms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/edx_ace/channel/django_email.py", line 68, in deliver
lms-worker_1                 |     mail.send()
lms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/django/core/mail/message.py", line 284, in send
lms-worker_1                 |     return self.get_connection(fail_silently).send_messages([self])
lms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/django/core/mail/backends/smtp.py", line 102, in send_messages
lms-worker_1                 |     new_conn_created = self.open()
lms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/django/core/mail/backends/smtp.py", line 69, in open
lms-worker_1                 |     self.connection.login(self.username, self.password)
lms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/smtplib.py", line 737, in login
lms-worker_1                 |     (code, resp) = self.auth(
lms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/smtplib.py", line 640, in auth
lms-worker_1                 |     (code, resp) = self.docmd("AUTH", mechanism + " " + response)
lms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/smtplib.py", line 430, in docmd
lms-worker_1                 |     return self.getreply()
lms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/smtplib.py", line 403, in getreply
lms-worker_1                 |     raise SMTPServerDisconnected("Connection unexpectedly closed")
lms-worker_1                 | smtplib.SMTPServerDisconnected: Connection unexpectedly closed
lms-worker_1                 | 2023-03-10 14:19:17,681 INFO 11 [edx.celery.task] [user None] [ip None] tasks.py:601 - BulkEmail ==> Status: Success, Task: a649930b-8a85-4ef8-a740-c006f6be6914, SubTask: 97285059-a3f0-409a-b3c6-b79ed9ce0a57, EmailId: 53, Recipient num: 91/119, Recipient UserId: 937

I attempted to fix this by editing the line of code here to include SMTPServerDisconnected, as suggested by Ghassan Maslamani on the forums. However that did not work, and the result was the same - partial send, but once the Server Disconnected error was received after 91 emails were sent, every subsequent email failed but incorrectly reported success.

I believe the correct behavior should be to queue up the email for attempted retry later.

If you'd like to try and reproduce this, I recommend using the Google SMTP servers, and sending emails to > 100 students. I am not just using gmail, but using a paid Google Workspace account. However, I expect the behavior to be similar when just using SMTP sending via any gmail account.

Start with lms/djangoapps/bulk_email/tasks.py around the referenced line and review the delivery path shown in edx_ace/channel/django_email.py. Reproduce the SMTP disconnect with a bulk send to more than 100 students, then verify that disconnects do not produce false success and that the email is queued for a later retry.