High volume email not woking properly with SES

Investigation: Goal Reminder Email Delivery Drop After Migration from Braze to SES

Description:

We are currently migrating the goal reminder email system from Braze to SES (using Django Email via edx-ace). After switching to SES, we observed a 50% drop in email delivery success rates, with no clear errors or logs explaining the issue. Despite thorough reviews of SES and edx-ace logs, the root cause remains unidentified.

Key Actions & Findings

1. Initial Approach:

   • Migrated emails to SES (edx-ace with Django Email backend).
   • Issue: 50% of emails failed to trigger events, with no error logs.
   • No errors related to SES were logged .
   • Added info logs are working as expected.

2. Troubleshooting Steps:

   • Bypassed edx-ace: Sent emails directly via boto3 (AWS SDK).
   • Result: Email events worked as expected, confirming the issue lies within the edx-ace/SES integration.
   • After this implementation, there might be some issue with django-ses package.
   • This implementation caused a drop in the sent emails by around 25%, which is still a significant drop.

Data Analysis

• Braze is 70% faster on average (0.180 seconds difference).
• SES shows no signs of throttling (350k weekly emails, no surge-related errors).

Open Questions

1. Should we prioritize further investigation into the edx-ace/SES integration issue?
   • I have no idea how to do that because there are no errors and info logs are all there.
   • AWS analytics are correct according to event data.
   • Some code performance improvements can be done in management command but they are not related to this issue.
2. Would creating a dedicated SES backend for edx-ace resolve the problem, based on the boto3 workaround? -
   • Yes, and we should create the new backend for SES.
   • At the moment we are using django-email -> django-ses flow .

Relevant Links

• Management command to send goal reminder email
• [Boto3 Workaround PR] (https://github.com/openedx/edx-platform/pull/36148)

Next Steps

We need feedback on whether to:

1. Dig deeper into the root cause of the edx-ace/SES integration issue.
2. Proceed with implementing a new SES backend for edx-ace based on the boto3 workaround.

Please share your thoughts and suggestions!

Start with lms/djangoapps/course_goals/management/commands/goal_reminder_email.py and the boto3 workaround in PR 36148. Compare the direct SES path with the edx-ace/Django email flow, then establish a reproducible delivery-success test and determine whether a dedicated SES backend resolves the observed drop.