Message Brokers

Summary: An intermediary that stores messages and delivers them asynchronously from a sender to a receiver.

Sources: chapter4, chapter11

Last updated: 2026-04-18

Message brokers (also known as message queues) are a common way to achieve asynchronous message-passing.

Advantages over Direct RPC

Improved Reliability: Acts as a buffer if the recipient is unavailable.
Redelivery: Automatically redelivers messages to a failed process.
Location Independence: Senders don’t need to know the IP address of the recipient.
Fan-out: One message can be delivered to multiple recipients.
Decoupling: Decouples the sender from the recipient.

As discussed in chapter11, there are two main approaches to message brokers:

Focused on individual message delivery.

Workflow: The broker assigns individual messages to consumers. Once a consumer acknowledges a message, it is deleted from the broker.
Load Balancing: Multiple consumers can read from the same queue; the broker shares the work among them.
Implementations: RabbitMQ, ActiveMQ.

Focused on high throughput and replayability.

Workflow: The broker stores messages in an append-only append-only-log on disk. Consumers keep track of their position using an offset. Messages are not deleted after they are consumed.
Ordering: Provides strong ordering guarantees within a partition.
Implementations: Apache Kafka, Amazon Kinesis Streams.

Load Balancing: Each message is delivered to one of the consumers. Useful for expensive processing tasks.
Fan-out: Each message is delivered to all consumers. Provided by topic subscriptions or exchange bindings.