Outbox pattern
Improve reliability of your async communication.
The Outbox Pattern, also known as the "Message Broker Outbox," is a solution to overcome the challenges of distributed systems. With this approach, we can ensure data consistency in a microservices architecture without the need for complex distributed transactions.
Every design pattern must solve some kind of problem no?
Let’s take a look at a really simple scenario:
What will go wrong here?
At some point our message broker might/will fail. In that case the entity will be saved to the data store, however the other components that are interested in that event will be un-notified.
How does the outbox pattern deal with this kind of issues?
The general ideas of the pattern are:
We will no longer publish messages to our broker but rather store them in a data source.
A background job then will asynchronously pick up the messages, process and mark them as completed.
Any side effects?
With the outbox pattern we are effectively decoupling business logic from message processing. However there is something to keep in mind:
→ The Message relay might publish a message more than once, if the service cloud not mark the message as processed. As a result, a message consumer must be idempotent, perhaps by tracking the IDs of the messages that it has already processed.
How can this be implemented in a real world scenario?
Let’s take for this example the Notification System that I have been covering in my YouTube series on architecture. One place where we cloud set up an outbox is the Events Push service that is really the main entry point of the workflow which is essentially critical in the system.
— We don’t want to loose any incoming messages.
No code?
This is a newsletter that I want to keep short and concise this is why I have attached all the implementation details as a link to the github repository.
The code base can be found over in this repository : https://github.com/ByteRain0/ground.zero.modular.notifications
The general idea is really simple and the implementation might vary. In my case I used Entity Framework core for storing data in the database, since i’m already using it in other modules as well as Hangfire for running the background job.
This is just one of the ways you can implement it, you can use Dapper and Quartz as a lightweight alternative, however the design principles remain the same.
What to watch/read next?
Below you will find links to 3 series covering Integration testing in .NET, Minimal API’s and building a Modular monolith from beginner to advanced level.
Integration testing - from ground zero series