Messaging with RabbitMQ

02 Mar 2018

RabbitMQ is a robust message broker that can be used to implement different messaging patterns. Even though there is an exellent tutorial available (using different languages and frameworks) it can be a bit difficult to get into the concepts. In this post I want to show some different paradigms that can be implemented with RabbitMQ and why I struggled with some of the concepts.

Sending and receiving using queues

The easiest thing to do is to use a queue for sending the messages and having a consumer that reads from the same queue.

Nothing stops you from having multiple consumers, that each can process messages from the queue. After a message is consumed it is gone from the queue.

This is especially well suited when having tasks that need to be executed and it doesn't matter which of the consumer processes the task.

Publish/Subscribe

Not all use cases are in the way that you just want to consume a message once. Often you want to have multiple consumers that should all process all the messages. One example can be storing objects in different data stores (e.g. a search index and the database), another one domain events like an order that has been submitted and should be processed by the order management system and the inventory system. This calls for a publish/subscribe mechanism and of course RabbitMQ has you covered for this.

The biggest difference compared to using a queue alone is that in this case the producer doesn't write to the queue directly anymore. An instance of what is called an Exchange accepts messages and forwards them to one or more queues.

To have a classic publish/subscribe model you would use a FanoutExchange which forwards the messages to one or more queues. To connect exchange and queue you are declaring a binding, in this case that all messages for a certain exchange should be forwarded to a certain queue.

Each consumer reads messages from a dedicated queue. That also means that you will need one binding for each consumer that is listening.

With RabbitMQ it is possible to use queues that are automatically deleted when the consumer stops listening. This allows for very dynamic behaviour with consumers joining and leaving.

Looking at our first example you might be wondering why there are different ways to send messages, to queues and to exchanges. It turns out that sending to queues really is not possible. There is always a default exchange involved that just forwards the messages. This default exchange just takes the messages for a certain routing key (which is the queue name) and puts them in the queue with the same name.

Publish/Subscribe with filtering

Besides sending messages to all queues that are registered for an exchange it is also possible to filter them according to a routing key. All messages are being sent to an exchange and this exchange decides by looking at the routing key to which queues a message should be sent to.

If you want to do this for an exact match of the routing key this is being done by the DirectExchange.

When binding a DirectExchange to a queue you need to supply a routing key that will determine which routing keys will be considered for this queue. If you want to assign multiple routing keys you can just add multiple bindings for the same queue.

You can also supply wildcards for determining which routing keys should be used for a queue, this is done by using a TopicExchange that expects hierarchical routing keys.

Those two exchanges - Direct and Topic exchange - probably were confusing me the most at the beginning. When it comes to topic I am thinking about classic publish subscribe systems like when using a FanoutExchange. The exchange name is the topic clients are registering for. But here topic refers to a kind of routing on an existing exchange. Same with the DirectExchange: I would have expected a direct exchange to be similar to the first example where you send messages to queues directly. But a direct exchange in this case refers to direct routing and you always need to supply a routing key for this.

If you want to know more about the different kinds of exchanges head over to the tutorials on the RabbitMQ website.