InditexTech / InditexTech/scs-outbox
Evaluate batching outbox message persistence during transaction capture
- Dominant language
- Java
- Stars
- 19
- Forks
- 0
- Avg merge
- 9h 31m
- Merged PRs (30d)
- 17
Description
### Motivation
A single business transaction can produce multiple outbound events. At the moment, each captured event is persisted immediately through an individual repository operation.
This means that a transaction producing many events performs one database operation per event during the outbox capture phase. The individual operations participate in the same transaction, but the repeated database round trips can increase the time required to complete the business transaction unnecessarily.
The library should assess whether captured outbox messages can be accumulated during the current transaction and persisted using a batch operation, such as a JDBC batch insert or a MongoDB multi-document insert. This could reduce the number of database round trips and improve capture performance for transactions that produce many events.
### Acceptance criteria
- [ ] Measure the current capture performance and database operation count for transactions producing multiple outbox messages.
- [ ] Assess the feasibility of accumulating messages per transaction and persisting them using a batch operation.
- [ ] Evaluate the required changes to the `OutboxMessageRepository` contract and the capture flow.
- [ ] Evaluate a JDBC batch implementation for the supported databases, including PostgreSQL and MariaDB.
- [ ] Evaluate a MongoDB multi-document insert implementation.
- [ ] Preserve the atomicity of the business transaction and ensure that all captured messages are rolled back when the transaction rolls back.
- [ ] Preserve the order in which messages are captured.
- [ ] Define the behavior when the transaction contains only one captured message.
- [ ] Define the behavior for serialization errors, database errors, and partial batch failures.
- [ ] Define whether a configurable maximum batch size is required to control memory usage and database limits.
- [ ] Assess the impact on capture metrics, logging, and observability.
- [ ] Compare the performance, complexity, and operational trade-offs against the current individual persistence approach.
- [ ] Add unit and integration tests for multiple messages in one transaction, rollback, failures, ordering, JDBC, and MongoDB.
- [ ] Document the resulting behavior and any limitations if batching is implemented.
Contributor guide
Research direction
Start with the OutboxMessageRepository contract and the transaction capture flow, then measure current operation counts for transactions producing multiple messages. Compare JDBC batching for PostgreSQL and MariaDB with MongoDB multi-document inserts, and document the chosen behavior, transactional guarantees, ordering, failures, observability, and required unit and integration tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mariadb, mongodb, postgresql
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100