InditexTech / InditexTech/scs-outbox
Support compacting pending outbox messages before publishing
- Dominant language
- Java
- Stars
- 19
- Forks
- 0
- Avg merge
- 9h 31m
- Merged PRs (30d)
- 17
Description
### Motivation
Some applications generate multiple events for the same entity before the outbox publisher processes them. In these scenarios, publishing every intermediate state can produce unnecessary events because consumers are only interested in the latest state.
For example, when several pending messages target the same destination and use the same Kafka key, the publisher could retain and publish only the latest message for that key. Earlier messages would no longer provide useful information and should be removed from the outbox.
This behavior should be optional and must only apply when all messages in the target topic are compatible with this type of compaction. The library currently does not know the event payload type, so compacting messages representing different event types could result in data loss.
### Acceptance criteria
- [ ] Assess whether pending messages can be compacted before publishing.
- [ ] Define the configuration required to enable or disable compaction.
- [ ] Define how messages are identified as belonging to the same compaction group, including destination and Kafka key.
- [ ] When compaction is enabled, publish only the latest pending message for each compaction group.
- [ ] Remove compacted messages that are not published from the outbox.
- [ ] Preserve the current ordering and publishing behavior when compaction is disabled.
- [ ] Define the behavior for messages without a Kafka key.
- [ ] Add unit and integration tests covering compaction, deletion of discarded messages, ordering, retries, and publishing failures.
- [ ] Document the limitations and the requirement that messages in a compacted topic must be compatible.
Contributor guide
Research direction
Start by reading the existing outbox publishing and configuration behavior, then check how pending messages are stored and removed. Define compaction groups, keyless-message behavior, and failure/retry semantics before implementation; done means optional compaction preserves disabled behavior and the requested scenarios are covered by unit and integration tests and documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spring
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100