Discovery: Implement MVP of an outbox pattern for the event bus

A/Cs:

• Documentation (ideally a Confluence doc) that covers what we learned and what decisions were made.
  • ADR for outbox implementation (https://github.com/openedx/openedx-events/pull/292), communicated to and approved by other stakeholders
  • Outcome: https://docs.openedx.org/projects/openedx-events/en/latest/decisions/0015-outbox-pattern-and-production-modes.html
• A proposed set of tickets that cover the work needed to implement this pattern in an MVP format.

At this time, there is not a great story for recovery from event producing issues if the event bus (Kafka, Redis, etc.) were to temporarily go down.

A common solution for this is an Outbox Pattern, where event data is first sent to the database, and a separate process sends events from the outbox to the event broker, maintaining order. This ticket is for implementing an MVP of this pattern.

Notes/Questions:

• Another ticket, https://github.com/edx/edx-arch-experiments/issues/354, to document/implement a less ideal work around, but this fix is likely to be much more dependent on environment.
• To maintain order, the simplest implementation is to have a singleton process that reads from the outbox and produces the events.
  • What can this handle for typical load?
  • What can this handle for a large incident?
  • When would we need to invest in being able to have multiple processors (e.g. that handle different topics)?
  • Would we need a capability to move or keep certain topics with large load and less essential events off the outbox?
• When do we delete events from the outbox?
• Where is the code for this going to live? Do we want this to be in a library?
• Note: there are many articles we can review to learn from others.

No files, tests, or entry points are named in the issue. Start with the linked ADR and stakeholder discussion, then review the existing event-bus implementation and the related workaround issue. Done means proposing a set of implementation tickets that addresses the MVP scope and records the unresolved operational decisions.