Explained with a diagram · Microservices
Outbox pattern: how the transactional outbox works, with a diagram
In short
The outbox pattern is a way for a service to update its database and publish a message without a distributed transaction. The service stores the outgoing message in an outbox table as part of the same local transaction that changes its business data, and a separate message relay then reads the outbox and sends each message to the message broker. A message is therefore sent if, and only if, the transaction commits. It is also known as the transactional outbox pattern.
Updated · by the ChartCraft team · one of the patterns and models explained with a diagram
What does the outbox pattern look like as a diagram?
The diagram has one service, one database and one broker. The service writes to two tables, its business table and the outbox table, inside a single transaction. A message relay moves rows from the outbox to the message broker, and consumers read from the broker.
Mermaid code for this diagram
flowchart TD
subgraph db["Order service database (one local transaction)"]
orders[("ORDERS table (business data)")]
outbox[("OUTBOX table (messages to send)")]
end
sender["Order service (sender)"]
relay["Message relay"]
broker["Message broker"]
consumer["Consumer service (idempotent)"]
sender -->|"1. insert order"| orders
sender -->|"2. insert message"| outbox
outbox -->|"3. read new rows"| relay
relay -->|"4. publish"| broker
broker -->|"5. deliver"| consumer
Paste it into any Mermaid renderer, or check and export it with the Mermaid syntax checker and Mermaid to PNG.
What are the parts of the outbox pattern?
The pattern has four participants: the sender, its database, the message outbox inside that database, and the message relay. The broker and the consumers are the systems the messages are for.
| Part of the diagram | What it is | What it does |
|---|---|---|
| Order service (sender) | The service that sends the message | Handles a command, changes its business data and writes the message to the outbox in the same transaction. It never calls the broker directly. |
| ORDERS table (business data) | The business tables | Hold the entities the command creates or updates, here a new order. |
| OUTBOX table (messages to send) | The message outbox | In a relational database, a table that stores the messages to be sent. In a NoSQL database, the outbox is a property of each record instead. |
| Message relay | A separate process | Sends the messages stored in the outbox to the message broker, in the order the service wrote them. |
| Message broker | The messaging system | Receives each message from the relay and delivers it to subscribers. |
| Consumer service (idempotent) | A service that reacts to the message | Must be idempotent, because the relay can publish the same message more than once. |
How does the outbox pattern work, step by step?
The service does two inserts in one transaction, and the relay does the publishing later. Nothing is sent to the broker inside the request itself.
- A command arrives, for example place order.
- The service opens a database transaction and inserts or updates its business rows.
- In the same transaction it inserts a row describing the message, such as
OrderCreated, into the outbox table. - The transaction commits. If it rolls back, the business change and the message disappear together.
- The message relay finds the new outbox row and publishes the message to the broker.
- Consumers receive the message and ignore it if they have already processed its ID.
How does the message relay read the outbox?
There are two ways to implement the relay: a polling publisher, which queries the outbox table for unpublished rows, and transaction log tailing, which reads the database's transaction log and publishes each message inserted into the outbox.
| Polling publisher | Transaction log tailing | |
|---|---|---|
| How it works | Publishes messages by polling the outbox table | Tails the transaction log, for example the MySQL binlog, the Postgres WAL or DynamoDB table streams |
| Strength | Works with any SQL database | No two-phase commit and guaranteed to be accurate |
| Weakness | Tricky to publish events in order; not supported by every NoSQL database | Needs a database-specific solution, and avoiding duplicate publishing is tricky |
When should you use the outbox pattern?
Use it when one operation has to change the database and send a message, and the two must succeed or fail together. That is the usual situation for a service that takes part in a saga or publishes domain events.
Use it when
- A command must update business data and publish a message or event.
- A two-phase commit (2PC) across the database and the broker is unavailable or unwanted.
- Messages have to reach the broker in the order the service produced them.
Skip it when
- The operation changes data but nothing needs to be published.
- The service already uses event sourcing, which is an alternative solution to the same problem.
- Consumers cannot be made idempotent, since the relay may publish a message more than once.
What are common mistakes with the outbox pattern?
Most mistakes put the message outside the transaction or assume each message arrives exactly once.
- Publishing straight to the broker. Sending in the middle of a transaction is unreliable because the transaction may not commit, and sending after the commit is unreliable because the service can crash first.
- Writing the outbox row in a separate transaction. The message and the business change must commit together, or the pattern guarantees nothing.
- Assuming exactly-once delivery. The relay can crash after publishing but before recording that it did, and will publish again on restart. Track processed message IDs in the consumer.
- Losing message order. Events for the same aggregate must be published in the order their transactions ran; this is the tricky part of a polling relay.
- Forgetting the outbox write. The pattern relies on the developer remembering to store the message whenever the data changes.
How do you draw an outbox pattern diagram?
Start with the sender and its database, draw the business table and the outbox table inside one transaction boundary, then add the relay, the broker and one consumer, and number the arrows in the order they happen.
What do real event-driven architecture diagrams look like?
Live diagrams from the public ChartCraft gallery, each generated from a plain-English description. Open one to see the prompt that produced it.
Questions
What is the outbox pattern?
The outbox pattern is a way to update a database and publish a message reliably without a distributed transaction. The service writes the message to an outbox table in the same local transaction as its data change, and a separate message relay publishes it to the broker afterwards.
What problem does the outbox pattern solve?
It solves the problem of atomically updating a database and sending a message. Without it, a service that writes to the database and then calls the broker can crash between the two steps, leaving saved data with no message, or a message with no saved data.
Is the transactional outbox the same as the outbox pattern?
Yes. Transactional outbox is the name the pattern has in the microservices.io pattern catalogue; outbox pattern is the common short form.
Does the outbox pattern guarantee exactly-once delivery?
No. The message relay can publish a message more than once, for example if it crashes after publishing but before recording that it did. Consumers must be idempotent, typically by tracking the IDs of messages they have already processed.
What is the difference between a polling publisher and transaction log tailing?
They are the two ways to build the message relay. A polling publisher queries the outbox table for new rows and works with any SQL database. Transaction log tailing reads the database's transaction log, such as the MySQL binlog or Postgres WAL, and needs a database-specific solution.